Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

2026-06-30

AI Summary

Existing reconstruction-agnostic joint embedding objectives are prone to representation collapse and struggle to capture action-sensitive dynamics, thereby limiting planning and control performance. To address this, this work proposes Delta-JEPA, an end-to-end, pixel-reconstruction-free world model that introduces a Latent Difference Action Decoder (LDAD). LDAD reconstructs actions by modeling the displacement between consecutive observations in latent space, thereby strengthening the coupling between actions and latent dynamics. The method eliminates both pixel-level reconstruction and distribution-matching regularization, significantly outperforming JEPA and representation learning baselines across four visual continuous control tasks. Ablation studies confirm that the displacement-based decoding mechanism is crucial for enhancing action-conditioned responses and planning capabilities.

Abstract

Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations. We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments latent forward prediction with a Latent Difference Action Decoder (LDAD). Unlike inverse decoders that infer actions from concatenated endpoint embeddings, LDAD reconstructs the executed action from the latent displacement between consecutive observations. This displacement-level supervision directly regularizes transition geometry: adjacent embeddings cannot collapse without losing action information, and different actions are encouraged to induce distinguishable latent changes for rollout-based planning. Delta-JEPA uses only latent prediction and action reconstruction, avoiding pixel reconstruction and distribution-matching regularizers. Across four visual continuous-control tasks, Delta-JEPA improves planning over JEPA-based and representation-learning world model baselines. Ablations show that displacement-based action decoding is consistently more effective than endpoint concatenation, and action-sensitivity analyses show clearer action-conditioned latent responses. These results indicate that supervising latent differences is a simple and effective mechanism for collapse-resistant and action-sensitive world model learning.

Problem

• world models
• action sensitivity
• representation collapse
• latent dynamics
• joint-embedding

Innovation

• Delta-JEPA
• Latent Difference Action Decoder
• action-sensitive world models
• reconstruction-free learning
• latent dynamics