HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

2026-07-01

AI Summary

This work addresses the sensitivity of CLIP models to textual perturbations—such as sentence reordering, summarization, or omission—when processing long texts exceeding 77 tokens, a limitation stemming from their use of absolute positional encoding and pretraining on short sequences, which degrades image-text alignment. To overcome this, the authors propose the first hyperbolic fine-tuning framework for CLIP, which transfers alignment knowledge from Euclidean to hyperbolic space via cross-manifold similarity distillation. The approach explicitly models the hierarchical part-whole semantic structure inherent in long texts by leveraging Einstein midpoint aggregation and hyperbolic entailment relations. Evaluated on multiple long-context cross-modal retrieval benchmarks, the method significantly outperforms existing approaches, achieving up to a 19.5% improvement in retrieval robustness under textual perturbations, and is successfully integrated into generative frameworks such as Stable Diffusion XL.

Abstract

CLIP (Contrastive Language-Image Pre-training) has become a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (>77 tokens) due to absolute positional encoding and pretraining on short captions. In long contexts, sentences are often reordered, summarized, or partially omitted. Although prior works extend CLIP with longer positional encodings, they often suffer from degraded image-text alignment under such text perturbations. We attribute this limitation to the Euclidean contrastive objective, which enforces strict one-to-one matching and lacks explicit mechanisms for modeling hierarchical relationships between global context and its constituent elements. To address this issue, we propose HyFL-CLIP, a hyperbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space via cross-manifold similarity distillation, leveraging its geometry to capture hierarchical and entailment relations. Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part-whole relationships via hyperbolic entailment with Einstein midpoint aggregation. Experiments on diverse benchmarks, including long-context cross-modal retrieval, cross-modal retrieval with caption perturbations, intra-modality retrieval, and short-text cross-modal retrieval, show that HyFL-CLIP achieves more robust long-context understanding. In particular, it yields up to 19.5% improvement in long-text cross-modal retrieval under textual perturbations over the best prior method. We also show HyFL-CLIP can be seamlessly integrated into other model frameworks by applying it to Stable Diffusion XL (SDXL).

Problem

• long-context understanding
• image-text alignment
• text perturbations
• hierarchical relationships
• CLIP

Innovation

• Hyperbolic Geometry
• CLIP Fine-tuning
• Long-Context Understanding
• Hierarchical Semantics
• Cross-Modal Retrieval