The Double-Ellipsoid Geometry of CLIP

2024-11-21

arXiv.org

AI Summary

This work investigates the geometric structure of unnormalized text and image embeddings in CLIP and its impact on contrastive learning. We discover that these embeddings reside on non-centered, linearly separable ellipsoidal shells—a geometry that implicitly encodes instance-level uncertainty. Building on this, we introduce, for the first time, a “consistency” metric to quantify embedding confidence and prove that the modality gap fundamentally reflects misalignment between the consistency distributions of image and text embeddings. Leveraging geometric analysis, cosine similarity modeling, and modality-wise mean estimation, we establish a theoretical framework for estimating embedding uncertainty. Our analysis quantitatively links false-negative frequency to ellipsoidal eccentricity, offering the first unified, geometry-based explanation for CLIP’s generalization and robustness.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning applications within a large variety of domains. We investigate the geometry of this embedding, which is still not well understood. We examine the raw unnormalized embedding and show that text and image reside on linearly separable ellipsoid shells, not centered at the origin. We explain the benefits of having this structure, allowing to better embed instances according to their uncertainty during contrastive training. Frequent concepts in the dataset yield more false negatives, inducing greater uncertainty. A new notion of conformity is introduced, which measures the average cosine similarity of an instance to any other instance within a representative data set. We show this measure can be accurately estimated by simply computing the cosine similarity to the modality mean vector. Furthermore, we find that CLIP's modality gap optimizes the matching of the conformity distributions of image and text.

Problem

• Investigates the geometry of unnormalized CLIP embeddings
• Explains benefits of ellipsoid shell structure for uncertainty embedding
• Introduces conformity measure optimizing image-text distribution matching

Innovation

• Analyzes unnormalized CLIP embedding geometry
• Introduces conformity measure via cosine similarity
• Optimizes image-text conformity distribution matching