Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Models

2026-06-19

AI Summary

This work addresses the limitations of imitation learning, which typically relies on high-quality, homogeneous demonstration data and struggles to generalize to scenarios with heterogeneous action spaces or absent action labels. To overcome this, the authors propose the Generalized Latent Action Model (GLAM), a framework that leverages environmental effect consistency to construct a shared latent action space across diverse data sources. GLAM employs a generative world model to automatically align heterogeneous demonstrations into a unified semantic representation, eliminating the need for manual design or explicit action annotations. A behavior cloning policy trained within this latent space enables cross-domain action transfer. Evaluated across five simulated and real-world manipulation tasks, GLAM achieves an average 48% improvement in task success rate over baseline methods, demonstrating substantially enhanced policy performance under data-scarce conditions.

Abstract

Imitation learning has emerged as a powerful paradigm for learning visuomotor policies, but its generalisation and stability are limited by the scale and quality of demonstration data needed. A promising direction is to leverage more abundant but heterogeneous data sources, which differ in action space and often lack action labels altogether. Existing co-training approaches that combine heterogeneous data sources rely on heuristic and hand-engineered alignment techniques. In contrast, we argue that action representations should be grounded in prediction: actions that produce the same effect on the environment should share the same representation, regardless of their sources. To this end, we instantiate this principle by using a grounded latent-action world model (GLAM), a pair of generative models with a shared latent action space across data sources that is grounded by predicting future observations consistently across sources. This latent action space is used to train downstream behavioural cloning (BC) policies which map observations to latent actions and decode them back to robot actions, providing a paradigm for learning from heterogeneous data. Empirically, we demonstrate that GLAM successfully learns an aligned latent action space that facilitates action transfer across data sources with and without action labels. Across five manipulation tasks in simulation and in the real world, GLAM-aligned policies significantly outperform BC baselines and prior latent-action methods, achieving an average of +48% improvement in task success rate with the same data-scarce setting. Videos and code are available at https://viccccciv.github.io/glam/.

Problem

• Imitation Learning
• Heterogeneous Demonstrations
• Latent Action
• World Models
• Action Grounding

Innovation

• grounded latent action
• heterogeneous imitation learning
• world model
• action representation alignment
• behavioral cloning