Learning by Analogy: A Causal Framework for Composition Generalization

2025-12-11

AI Summary

This work addresses compositional generalization—the ability of models to generalize to unseen combinations of learned concepts through novel recombinations. To overcome the limitations of existing methods—namely, their lack of causal interpretability and structural identifiability—we propose a theoretical framework grounded in causal modularity and the principle of minimal change: high-level concepts are decomposed into low-level causal modules that can be recombined across contexts, and we prove that hierarchical generative structures are uniquely identifiable under weakly supervised multimodal observations (e.g., text–image pairs). Our method integrates causal modeling, structural identifiability analysis, and causally disentangled representation learning. Empirically, it achieves significant improvements over state-of-the-art methods across multiple compositional generalization benchmarks. Crucially, it moves beyond traditional additive representation assumptions and establishes, for the first time, a verifiable causal foundation for modeling complex conceptual relationships.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Compositional generalization -- the ability to understand and generate novel combinations of learned concepts -- enables models to extend their capabilities beyond limited experiences. While effective, the data structures and principles that enable this crucial capability remain poorly understood. We propose that compositional generalization fundamentally requires decomposing high-level concepts into basic, low-level concepts that can be recombined across similar contexts, similar to how humans draw analogies between concepts. For example, someone who has never seen a peacock eating rice can envision this scene by relating it to their previous observations of a chicken eating rice. In this work, we formalize these intuitive processes using principles of causal modularity and minimal changes. We introduce a hierarchical data-generating process that naturally encodes different levels of concepts and their interaction mechanisms. Theoretically, we demonstrate that this approach enables compositional generalization supporting complex relations between composed concepts, advancing beyond prior work that assumes simpler interactions like additive effects. Critically, we also prove that this latent hierarchical structure is provably recoverable (identifiable) from observable data like text-image pairs, a necessary step for learning such a generative process. To validate our theory, we apply insights from our theoretical framework and achieve significant improvements on benchmark datasets.

Problem

• Develops a causal framework for compositional generalization
• Identifies latent hierarchical structures from observable data
• Improves model performance on benchmark datasets

Innovation

• Hierarchical data-generating process for concept encoding
• Causal modularity principles enabling compositional generalization
• Provably recoverable latent structure from observable data