Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models

2026-05-07

AI Summary

This work investigates whether reconstructive latent spaces—optimized for pixel-level reconstruction—or semantic latent spaces—aligned with high-level semantics—are more effective for policy learning in latent diffusion model–based robotic world models. Using a unified protocol, we train action-conditional latent diffusion models (LDMs) on the BridgeV2 dataset and systematically evaluate six encoders, including reconstructive variants (e.g., VAE, Cosmos) and semantic variants (e.g., V-JEPA 2.1, Web-DINO, SigLIP 2). Through a three-axis assessment encompassing visual fidelity, planning and downstream policy performance, and latent representation quality, we demonstrate for the first time in high-dimensional compressed spaces that, despite superior reconstruction accuracy of reconstructive encoders, semantic encoders—particularly V-JEPA 2.1—significantly enhance policy performance, revealing that visual fidelity alone is insufficient to guide world model design.

Abstract

World model-based policy evaluation is a practical proxy for testing real-world robot control by rolling out candidate actions in action-conditioned video diffusion models. As these models increasingly adopt latent diffusion modeling (LDM), choosing the right latent space becomes critical. While the status quo uses autoencoding latent spaces like VAEs that are primarily trained for pixel reconstruction, recent work suggests benefits from pretrained encoders with representation-aligned semantic latent spaces. We systematically evaluate these latent spaces for action-conditioned LDM by comparing six reconstruction and semantic encoders to train world model variants under a fixed protocol on BridgeV2 dataset, and show effective world model training in high-dimensional representation spaces with and without dimension compression. We then propose three axes to assess robotic world model performance: visual fidelity, planning and downstream policy performance, and latent representation quality. Our results show visual fidelity alone is insufficient for world model selection. While reconstruction encoders like VAE and Cosmos achieve strong pixel-level scores, semantic encoders such as V-JEPA 2.1 (strongest overall on policy), Web-DINO, and SigLIP 2 generally excel across the other two axes at all model scales. Our study advocates semantic latent space as stronger foundation for policy-relevant robotics diffusion world models.

Problem

• latent space
• world models
• robotic control
• semantic representation
• diffusion models

Innovation

• semantic latent space
• action-conditioned diffusion models
• world model evaluation
• representation learning
• robotic policy