Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition

2025-02-19

AI Summary

This study addresses the problem that vision-language models (VLMs) exhibit degraded out-of-distribution generalization in few-shot adaptation due to overreliance on spurious visual attributes—those co-occurring with categories but lacking semantic essence. To mitigate this, we propose the Spurious Attribute Probing and Shielding (SAP/SAS) framework. SAP systematically identifies such spurious attributes via a causal-inspired, vision-attribute-decoupled probing module; SAS then suppresses their influence through a gradient-based shielding mechanism, both integrated within a parameter-efficient fine-tuning (PEFT) paradigm. Our approach is the first to formally characterize VLMs’ bias toward spurious correlations. Evaluated across 11 benchmarks and three distribution shift settings—including domain, style, and object-background shifts—it significantly improves few-shot generalization accuracy while preserving original downstream task performance, establishing new state-of-the-art results in few-shot vision-language recognition.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization. Recent research has utilized low-level concepts such as visual attributes to enhance generalization. However, this study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, termed spuriously correlated attributes. This biased nature of VLMs results in poor generalization. To address this, 1) we first propose Spurious Attribute Probing (SAP), identifying and filtering out these problematic attributes to significantly enhance the generalization of existing attribute-based methods; 2) We introduce Spurious Attribute Shielding (SAS), a plug-and-play module that mitigates the influence of these attributes on prediction, seamlessly integrating into various Parameter-Efficient Fine-Tuning (PEFT) methods. In experiments, SAP and SAS significantly enhance accuracy on distribution shifts across 11 datasets and 3 generalization tasks without compromising downstream performance, establishing a new state-of-the-art benchmark.

Problem

• VLMs rely on spurious attributes for decisions.
• SAP filters out spurious attributes for generalization.
• SAS mitigates spurious attribute influence in predictions.

Innovation

• Identifying spurious attributes
• Filtering problematic attributes
• Mitigating attribute influence