⚓ CASPA: Graph-Structured Concept Anchors for Modality-Agnostic Adaptation in Vision–Language Models

Abstract

Recent advances in vision–language models (VLMs) have revealed both the promise and the rigidity of large-scale pretraining. Despite their impressive zero-shot generalization, existing adaptation paradigms—whether prompt tuning, adapter injection, or fine-tuning—remain class-specific, modality-biased, and structure-agnostic. However, these design choices limit reasoning-level transfer across tasks. To this end, we rethink adaptation as a shared conceptual structure rather than a per-class specialization. We propose CASPA (Concept-Anchored Semantic Prompt Adapter), a dual-anchor semantic adapter that jointly learns shared text and image anchors as a bidirectional conceptual interface between modalities. Each class learns a soft association distribution over these anchors, producing compositional representations which enable parameter sharing and semantic reuse. To further align visual and textual reasoning spaces, CASPA employs Semantic Cross-Consistency Regularization (S-XCR), enforcing geometric and semantic agreement between text- and image- conditioned anchor mixtures. CASPA, therefore, provides a structurally constrained alternative to class-conditional prompt parameterization while keeping the CLIP backbone frozen. Evaluated across four adaptation regimes—Base-to-Novel generalization, cross-data transfer, few-shot learning, and backbone-agnostic evaluation—on eleven diverse visual recognition datasets, CASPA matches or outperforms state-of-the-art methods.

CASPA Snapshot

Overview of CASPA's base class adaptation

Novel transfer mechanism

Quantitative results of CASPA on benchmark datasets (Base to Novel generalization)

Acknowledgement

We would like to thank the Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, USA, for their support, and Jadavpur University, India, for providing computational resources that made this work possible.

BibTeX

@inproceedings{caspa2026,
title={CASPA: Graph-Structured Concept Anchors for Modality-Agnostic Adaptation in Vision–Language Models},
author={Chatterjee, Abhiroop and Ghosh, Susmita and Ghosh, Ashish and Ientilucci, Emmett},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2026}
}