Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)

2026-06-13

AI Summary

This work addresses the performance degradation of CLIP in online zero-shot learning caused by the mismatch between the label distribution of test data and that of the pretraining source domain. Framing this challenge as a domain adaptation problem, the study introduces a label shift-aware mechanism that leverages unlabeled test data to calibrate CLIP’s prediction distribution, aligning it with the target-domain label distribution. By integrating vision-language modeling, online domain adaptation, and unsupervised distribution matching, the proposed method significantly outperforms existing CLIP-based online zero-shot approaches across multiple benchmarks, demonstrating enhanced generalization capability and robustness.

Abstract

Vision-language models like Contrastive Language-Image Pre-Training (CLIP) have been extensively studied in data-scarce scenarios. A particularly challenging and realistic task in this area is online zero-shot learning with CLIP, where unknown test samples are predicted sequentially in random order by CLIP while keeping the feature extraction and model parameters fixed during the sequential inference phase. Most existing approaches in this setting address the problem by adapting representations online using incoming test samples, while neglecting the distribution of the data on which CLIP was initially trained. This mismatch can lead to degraded performance when the label distribution in the test data differs from that of the training domain. To address this gap, we propose Label Shift Aware (LSA), which formulates the online zero-shot classification task as a domain adaptation problem. Specifically, LSA adapts the predictions computed by CLIP, which was trained on an unknown source distribution, to a target distribution using only unlabeled test data, and applies label shift correction to mitigate the mismatch between the source and target domains. The extensive experiments across multiple datasets demonstrate that the proposed LSA consistently outperforms state-of-the-art online zero-shot learning methods based on CLIP.

Problem

• label shift
• online zero-shot learning
• CLIP
• domain adaptation
• distribution mismatch

Innovation

• Label Shift Correction
• Online Zero-shot Learning
• Domain Adaptation
• CLIP
• Vision-Language Models