Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

2026-02-27

AI Summary

Achieving compositional generalization in vision models under unseen compositional scenarios remains a fundamental challenge for foundation models. This work theoretically establishes, for the first time, that effective compositional generalization necessitates representations that are disentangled, transferable, and stable—properties that require conceptual components in the embedding space to be linearly decomposable and mutually orthogonal. We further derive a geometric constraint linking the number of concepts to the embedding dimensionality. Guided by this theory, we conduct empirical analyses on prominent models including CLIP, SigLIP, and DINO, revealing that their representations consistently exhibit partial linear factorization and low-rank near-orthogonal structures. Crucially, the degree of such structure strongly correlates with compositional generalization performance, thereby validating the proposed theoretical framework.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massive datasets, they still cover only a tiny fraction of the combinatorial space of possible inputs, raising the question of what structure representations must have to support generalization to unseen combinations. We formalize three desiderata for compositional generalization under standard training (divisibility, transferability, stability) and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. This provides theoretical grounding for the Linear Representation Hypothesis: the linear structure widely observed in neural representations is a necessary consequence of compositional generalization. We further derive dimension bounds linking the number of composable concepts to the embedding geometry. Empirically, we evaluate these predictions across modern vision models (CLIP, SigLIP, DINO) and find that representations exhibit partial linear factorization with low-rank, near-orthogonal per-concept factors, and that the degree of this structure correlates with compositional generalization on unseen combinations. As models continue to scale, these conditions predict the representational geometry they may converge to. Code is available at https://github.com/oshapio/necessary-compositionality.

Problem

• compositional generalization
• vision embedding models
• linear representations
• orthogonal representations
• representation geometry

Innovation

• compositional generalization
• linear representation
• orthogonal representations
• vision embedding models
• representation geometry