COLA: Context-aware Language-driven Test-time Adaptation

2025-09-22

AI Summary

Existing test-time adaptation (TTA) methods typically assume identical label spaces between source and target domains, limiting their applicability to realistic multi-target-domain scenarios with non-overlapping class sets. To address this, we propose COLA—a novel TTA framework that, for the first time, leverages pretrained vision-language models (e.g., CLIP) for label-space-agnostic multi-target-domain adaptation. Its core innovations include: (i) a lightweight task-aware adapter with contextual units that fuses task semantics, domain-specific features, and prior knowledge via residual connections; and (ii) a class-balanced pseudo-labeling strategy to mitigate label imbalance across target domains. COLA operates under frozen model parameters, enabling efficient cross-domain knowledge transfer. Extensive experiments on multiple TTA and class-generalization benchmarks demonstrate significant improvements in accuracy and robustness, while maintaining high parameter efficiency and strong generalizability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Test-time adaptation (TTA) has gained increasing popularity due to its efficacy in addressing ``distribution shift'' issue while simultaneously protecting data privacy. However, most prior methods assume that a paired source domain model and target domain sharing the same label space coexist, heavily limiting their applicability. In this paper, we investigate a more general source model capable of adaptation to multiple target domains without needing shared labels. This is achieved by using a pre-trained vision-language model (VLM), egno, CLIP, that can recognize images through matching with class descriptions. While the zero-shot performance of VLMs is impressive, they struggle to effectively capture the distinctive attributes of a target domain. To that end, we propose a novel method -- Context-aware Language-driven TTA (COLA). The proposed method incorporates a lightweight context-aware module that consists of three key components: a task-aware adapter, a context-aware unit, and a residual connection unit for exploring task-specific knowledge, domain-specific knowledge from the VLM and prior knowledge of the VLM, respectively. It is worth noting that the context-aware module can be seamlessly integrated into a frozen VLM, ensuring both minimal effort and parameter efficiency. Additionally, we introduce a Class-Balanced Pseudo-labeling (CBPL) strategy to mitigate the adverse effects caused by class imbalance. We demonstrate the effectiveness of our method not only in TTA scenarios but also in class generalisation tasks. The source code is available at https://github.com/NUDT-Bai-Group/COLA-TTA.

Problem

• Adapts pre-trained vision-language models to multiple target domains without shared label spaces
• Addresses VLMs' inability to capture distinctive attributes of target domains effectively
• Mitigates class imbalance issues during test-time adaptation through balanced pseudo-labeling

Innovation

• Lightweight context-aware module with three components
• Seamlessly integrates into frozen vision-language model
• Class-balanced pseudo-labeling strategy mitigates imbalance