Semantic Relation-Enhanced CLIP Adapter for Domain Adaptive Zero-Shot Learning

2025-10-21

AI Summary

In domain-adaptive zero-shot learning (DAZSL), existing methods struggle to jointly achieve cross-domain transfer and cross-category generalization—especially under high annotation costs and scarce target-domain data—largely failing to harness the semantic generalization capacity of vision-language models (e.g., CLIP), resulting in inefficient knowledge transfer and degraded cross-modal alignment during fine-tuning. To address this, we introduce CLIP into the DAZSL framework for the first time, proposing a semantic relational structure loss and a cross-modal alignment preservation strategy. These jointly model the semantic topology among categories while enforcing consistency between visual and textual embedding spaces. Our approach achieves significant improvements over state-of-the-art methods on the I2AwA and I2WebV benchmarks, demonstrating superior effectiveness and robustness in joint generalization to both unseen classes and the target domain.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The high cost of data annotation has spurred research on training deep learning models in data-limited scenarios. Existing paradigms, however, fail to balance cross-domain transfer and cross-category generalization, giving rise to the demand for Domain-Adaptive Zero-Shot Learning (DAZSL). Although vision-language models (e.g., CLIP) have inherent advantages in the DAZSL field, current studies do not fully exploit their potential. Applying CLIP to DAZSL faces two core challenges: inefficient cross-category knowledge transfer due to the lack of semantic relation guidance, and degraded cross-modal alignment during target domain fine-tuning. To address these issues, we propose a Semantic Relation-Enhanced CLIP (SRE-CLIP) Adapter framework, integrating a Semantic Relation Structure Loss and a Cross-Modal Alignment Retention Strategy. As the first CLIP-based DAZSL method, SRE-CLIP achieves state-of-the-art performance on the I2AwA and I2WebV benchmarks, significantly outperforming existing approaches.

Problem

• Addresses inefficient cross-category knowledge transfer in domain adaptation
• Solves degraded cross-modal alignment during target domain fine-tuning
• Enhances CLIP's performance for domain-adaptive zero-shot learning scenarios

Innovation

• Semantic Relation Structure Loss enhances knowledge transfer
• Cross-Modal Alignment Retention Strategy maintains feature consistency
• Adapter framework integrates semantic relations with CLIP model