Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning

2026-02-24

AI Summary

This work addresses the well-documented limitation of vision-language models such as CLIP in accurately interpreting negation, often misaligning negative textual descriptions with their corresponding positive visual instances. To overcome this, the authors propose CLIPGlasses, a plug-and-play framework that enhances CLIP’s understanding of negation without fine-tuning its text encoder. CLIPGlasses employs a two-stage architecture: a Lens module that disentangles negation semantics and a Frame module that models context-aware repulsion strength, jointly introducing a penalty mechanism into the similarity computation. The method maintains strong in-domain performance while significantly outperforming existing approaches in cross-domain generalization, with particularly pronounced advantages in low-resource settings.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-Language Models (VLMs) like CLIP struggle to understand negation, often embedding affirmatives and negatives similarly (e.g., matching"no dog"with dog images). Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting. In this work, we propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions. CLIPGlasses adopts a dual-stage design: a Lens module disentangles negated semantics from text embeddings, and a Frame module predicts context-aware repulsion strength, which is integrated into a modified similarity computation to penalize alignment with negated semantics, thereby reducing false positive matches. Experiments show that CLIP equipped with CLIPGlasses achieves competitive in-domain performance and outperforms state-of-the-art methods in cross-domain generalization. Its superiority is especially evident under low-resource conditions, indicating stronger robustness across domains.

Problem

• negation understanding
• vision-language models
• CLIP
• visual descriptions
• semantic disentanglement

Innovation

• negation understanding
• vision-language models
• CLIP
• plug-and-play framework
• cross-domain generalization