Intervening to learn and compose disentangled representations

2025-07-07

AI Summary

This work addresses the fundamental tension between expressive power and disentanglement of latent structure in generative models. We propose a novel method to learn identifiable disentangled representations without compromising expressivity. Our approach augments the standard encoder-decoder architecture with a decoder-only module that implicitly inverts the encoder’s linear representation, coupled with a dynamically scheduled training mechanism grounded in causal intervention to enable compact joint modeling across multiple contexts. Theoretically, we establish—for the first time—the identifiability of structured latent representations in nonlinear generative models under mild assumptions. Empirically, our method achieves significant improvements in disentanglement quality, enabling cross-concept compositionality and out-of-distribution generalization. The resulting framework jointly delivers strong representational capacity, structural interpretability of latent factors, and robust generalization—without sacrificing fidelity or flexibility.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

In designing generative models, it is commonly believed that in order to learn useful latent structure, we face a fundamental tension between expressivity and structure. In this paper we challenge this view by proposing a new approach to training arbitrarily expressive generative models that simultaneously learn disentangled latent structure. This is accomplished by adding a simple decoder-only module to the head of an existing decoder block that can be arbitrarily complex. The module learns to process concept information by implicitly inverting linear representations from an encoder. Inspired by the notion of intervention in causal graphical models, our module selectively modifies its architecture during training, allowing it to learn a compact joint model over different contexts. We show how adding this module leads to disentangled representations that can be composed for out-of-distribution generation. To further validate our proposed approach, we prove a new identifiability result that extends existing work on identifying structured representations in nonlinear models.

Problem

• Balancing expressivity and structure in generative models
• Learning disentangled latent representations effectively
• Enabling out-of-distribution generation via composed representations

Innovation

• Adding decoder-only module for disentangled learning
• Selective architecture modification via intervention
• Proving identifiability for nonlinear structured representations