DiLA: Disentangled Latent Action World Models

2026-05-15

AI Summary

This work addresses the fundamental trade-off between action abstraction and generation fidelity in latent action models. By introducing a content-structure disentanglement mechanism, the approach leverages a predictive bottleneck to drive latent action learning, separating spatial layout (structure) from visual details (content). A dual-path architecture is designed to enable their co-evolution, thereby constructing a continuous and semantically structured latent action space. Integrating self-supervised world modeling with latent action inference, the method achieves state-of-the-art performance across multiple tasks—including video generation quality, action transfer, visual planning, and manifold interpretability—effectively unifying high-level abstraction with high-fidelity generation.

Abstract

Latent Action Models (LAMs) enable the learning of world models from unlabeled video by inferring abstract actions between consecutive frames. However, LAMs face a fundamental trade-off between action abstraction and generation fidelity. Existing methods typically circumvent this issue by using two-stage training with pre-trained world models or by limiting predictions to optical flow. In this paper, we introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement. Our key insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement, compelling the model to distill spatial layouts into the structure pathway while offloading visual details to a separate content pathway for generation. This synergy yields a continuous, semantically structured latent action space without compromising generative quality. DiLA achieves superior results in video generation quality, action transfer, visual planning, and manifold interpretability. These findings establish DiLA as a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning.

Problem

• Latent Action Models
• action abstraction
• generation fidelity
• world models
• disentanglement

Innovation

• Disentangled Representation
• Latent Action Models
• World Models
• Content-Structure Disentanglement
• Self-supervised Learning