Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models

2025-10-01

Pattern Recognition

AI Summary

Existing vision-language models suffer from poor generalization in zero-shot tasks—particularly against unseen classes—due to two key limitations: (1) biases introduced by image augmentations are not explicitly modeled, causing prompts to overfit augmentation artifacts; and (2) the absence of semantic guidance hinders prompt focus on intrinsic visual semantics. To address this, we propose the first prompt learning framework that *decouples augmentation bias*, jointly leveraging causal intervention and contrastive learning to explicitly separate semantic features from augmentation-related ones, while embedding a learnable prompt network. Our method requires no additional annotations and achieves significant performance gains across multiple zero-shot and cross-domain benchmarks. Notably, it demonstrates superior generalization under low-data and out-of-distribution settings. This work establishes a novel paradigm for robust, semantics-aware prompt learning in vision-language models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Problem

• Decoupling augmentation bias from semantic representations in prompt learning
• Improving generalization of vision-language models to unseen categories
• Addressing limitations of existing prompt learning methods like CoCoOp

Innovation

• AAPL introduces adversarial token embeddings for prompts
• Decouples augmentation bias from semantic representations
• Focuses prompts on visually discriminative category features