On the modality gap and the contrastive loss in multi-modal representation learning

2026-07-12

AI Summary

This work addresses the persistent modality gap between image and text embeddings in CLIP-style dual-encoder contrastive learning, despite their projection into a shared space. The authors identify this issue as stemming from mode collapse of the InfoNCE loss under low temperature settings. To mitigate this, they propose xNCE, a novel contrastive learning approach that jointly incorporates both cross-modal and intra-modal negative pairs. This strategy effectively narrows the modality gap while preserving the discriminative geometric structure of the embedding space. Experimental results demonstrate that xNCE maintains competitive performance on image-text retrieval benchmarks such as MS-COCO and achieves significant improvements in zero-shot classification accuracy across multiple standard datasets.

Abstract

We study the modality gap in CLIP-style dual-encoder contrastive learning, where image and text embeddings remain misaligned despite being trained in a shared space. We argue that the gap is induced by a failure of the InfoNCE formulation with independent encoders. We conduct a uni-modal experiment with two independent encoders and identical initialization conditions and find that InfoNCE actively generates a gap at low temperatures. We provide a theoretical analysis of this phenomenon and show that the modality gap is indeed a mode-failure of InfoNCE, but only at low temperatures. We propose a simple modification called xNCE, which uses intermodal as well as intra-modality negative contrastive pairs. xNCE matches retrieval performance on MS-COCO while consistently reducing the gap even at low temperatures. Notably, xNCE improves zero-shot classification over the InfoNCE baseline across all benchmarks, whereas high-temperature InfoNCE and regularized InfoNCE both fail to do so, demonstrating that xNCE reduces the modality gap without sacrificing the discriminative geometry needed for transfer.

Problem

• modality gap
• contrastive learning
• multi-modal representation
• InfoNCE
• embedding alignment

Innovation

• modality gap
• InfoNCE
• xNCE
• contrastive learning
• zero-shot classification