The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model

2026-07-15

AI Summary

This work addresses the limitations of existing probabilistic models in the CLIP latent space, which typically rely on Gaussian assumptions that fail to capture its hyperspherical geometry and multimodal semantic structure. To overcome this, the paper introduces, for the first time, a mixture of von Mises–Fisher distributions (MovMF) combined with an EM algorithm to construct a geometrically consistent semantic mixture framework on the unit hypersphere, where each component corresponds to an interpretable semantic concept. By moving beyond the restrictive isotropic Gaussian assumption, the proposed method significantly improves performance in detecting long-tailed and out-of-distribution samples while enabling sparse, interpretable semantic decomposition of embedding vectors.

Abstract

Contrastive Language-Image Pretraining (CLIP) representations form a semantic embedding space governed by cosine similarity, reflecting an intrinsic hyperspherical geometry. However, existing probabilistic interpretations typically rely on Gaussian assumptions, which fail to capture this directional and multimodal structure. We propose a principled density model for the CLIP latent space based on Mixtures of von Mises-Fisher (MovMF) distributions defined on the unit hypersphere. Using the Expectation-Maximization (EM) algorithm, we efficiently learn a probabilistic model in which each mixture component corresponds to a coherent semantic concept. This formulation yields a closed-form likelihood naturally aligned with hyperspherical geometry, enabling accurate and interpretable density estimation. Empirically, our model significantly improves long-tailed and out-of-distribution detection and provides a natural semantic decomposition, representing each embedding as a sparse probabilistic combination of interpretable concepts. These results suggest that CLIP latent space is more faithfully characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian, establishing a simple and geometrically consistent probabilistic framework for modeling and understanding multimodal representations. Project page is available at https://xiaoyuzhizi.github.io/movmf-clip/.

Problem

• CLIP latent space
• hyperspherical geometry
• semantic mixture
• von Mises-Fisher distribution
• multimodal representations

Innovation

• hyperspherical geometry
• von Mises-Fisher mixture
• CLIP latent space
• semantic decomposition
• density estimation