A Complexity-Based Theory of Compositionality

2024-10-18

arXiv.org

AI Summary

Existing research lacks a unified, quantifiable theory of representation compositionality, hindering its modeling and application in human cognition and AI. Method: Grounded in algorithmic information theory, this work introduces the first rigorous, general, and measurable formal definition of compositionality—characterized by three constraints: expressive power, symbolic decomposability, and semantic mapping conciseness. We integrate symbolic sequence modeling with deep neural networks to estimate compositionality and systematically validate our framework on both synthetic and real-world multimodal datasets. Although compositionality estimation is NP-hard, it remains approximable in practice. Contribution/Results: This work establishes the first mathematically consistent quantitative framework for compositionality, enabling principled measurement and analysis. It provides foundational tools for developing next-generation AI models endowed with compositional reasoning capabilities.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Compositionality is believed to be fundamental to intelligence. In humans, it underlies the structure of thought, language, and higher-level reasoning. In AI, compositional representations can enable a powerful form of out-of-distribution generalization, in which a model systematically adapts to novel combinations of known concepts. However, while we have strong intuitions about what compositionality is, there currently exists no formal definition for it that is measurable and mathematical. Here, we propose such a definition, which we call representational compositionality, that accounts for and extends our intuitions about compositionality. The definition is conceptually simple, quantitative, grounded in algorithmic information theory, and applicable to any representation. Intuitively, representational compositionality states that a compositional representation satisfies three properties. First, it must be expressive. Second, it must be possible to re-describe the representation as a function of discrete symbolic sequences with re-combinable parts, analogous to sentences in natural language. Third, the function that relates these symbolic sequences to the representation, analogous to semantics in natural language, must be simple. Through experiments on both synthetic and real world data, we validate our definition of compositionality and show how it unifies disparate intuitions from across the literature in both AI and cognitive science. We also show that representational compositionality, while theoretically intractable, can be readily estimated using standard deep learning tools. Our definition has the potential to inspire the design of novel, theoretically-driven models that better capture the mechanisms of compositional thought.

Problem

• Complexity Quantification
• Human Intelligence
• Artificial Intelligence

Innovation

• Compositional Theory
• Modular Reorganization
• Complexity Quantification