Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

2026-09-07

AI Summary

该研究通过变换感知的提示条件和重新校准的对比损失，解决视觉-语言模型在迁移学习中忽视数据特征的问题，提高模型泛化能力。

Abstract

Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models'ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.

Problem

• transfer learning
• vision-language models
• data characteristics
• independent transformation
• same-class samples

Innovation

• transformation-aware prompt conditioning
• re-calibrated contrastive loss
• positive-gradient dilution