HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models

2026-06-22

AI Summary

Current vision-language models exhibit insufficient sensitivity to negation semantics, are prone to interference from misleading textual cues, and often suffer from catastrophic forgetting of pre-existing knowledge when fine-tuned on negation data. To address these limitations, this work proposes an embedding space restructuring approach that explicitly models the semantic notion of what an image is *not*, while preserving the structural integrity of pre-trained representations. By leveraging hyperbolic geometry to capture semantic hierarchies and asymmetries, and introducing an angular triplet loss to systematically disentangle positive and negative descriptions, the method achieves robust negation reasoning with only a few negation examples. The approach is compatible with various CLIP variants, significantly outperforms baselines on NegBench, and maintains or even surpasses original performance on standard image classification and image-text retrieval benchmarks.

Abstract

Vision-Language Models (VLMs) are typically pre-trained on large-scale image-text datasets to capture semantic correspondences between visual content and natural language. However, they remain surprisingly brittle to negation: models often rely on shallow word co-occurrence and are easily distracted by misleading or irrelevant textual cues, even when their overall retrieval or classification performance is strong. Moreover, directly finetuning on negation data can interfere with previously acquired knowledge, causing noticeable degradation on standard vision-language benchmarks. To tackle these issues, this work introduces HANCLIP (Hyperbolic + Angular + Negation), a family of VLMs that explicitly restructures the embedding space to encode "what an image is not" alongside "what it is." HANCLIP is trained on a compact set of 20,000 image-text quadruplets and combines a hyperbolic formulation, which models hierarchical semantic relations and asymmetries, with an angular triplet objective that drives systematic separation between negated descriptions and their corresponding positives. This geometry-aware design strengthens negation sensitivity while preserving the global structure of pretrained representations, rather than overwriting them. Extensive experiments across multiple vision-language tasks show that HANCLIP delivers consistent gains on the negation-focused NegBench benchmark, while maintaining competitive or improved performance on standard classification and image-text retrieval benchmarks. The framework is model-agnostic and can be plugged into CLIP, LongCLIP, SmartCLIP, and HiMo-CLIP without large-scale retraining, demonstrating that a carefully designed geometric objective can substantially extend the reasoning capabilities of existing VLMs using only modest additional data.

Problem

• negation
• vision-language models
• semantic robustness
• embedding space
• model brittleness

Innovation

• Hyperbolic Geometry
• Angular Triplet Objective
• Negation Reasoning
• Vision-Language Models
• Embedding Space Restructuring