Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model

2026-06-25

AI Summary

Existing action-conditioned world models struggle to capture fine-grained action effects in high-DoF dexterous manipulation due to global compression of action sequences. This work proposes DexAC-WM, a framework that treats action conditioning as a structured process: it preserves dimensional semantics through action tokens, aligns actions with visual dynamics via local refinement and global modulation, and incorporates a semantic branch to inject object-scene priors. By uniquely integrating structured action modeling with high-level semantic guidance, DexAC-WM breaks away from conventional global-compression paradigms. Evaluated on EgoDex and EgoVerse, the method significantly improves FID, FVD, and PCK metrics, enhancing both temporal realism in video prediction and action-following consistency, while demonstrating strong scalability across diverse backbone architectures.

Abstract

Recent advances in action-conditioned world models show promising progress in modeling complex interactions and forecasting future states under diverse action sequences. While these models are often driven by stronger visual representations and model capacity, action conditioning itself remains underexplored. Most existing approaches compress the entire action sequence into a single representation, which works well for low-DoF control but becomes less reliable in high-DoF scenarios. We observe that high-DoF dexterous actions are inherently heterogeneous, spanning multiple orders of magnitude, where large-scale motions coexist with subtle but important signals. When uniformly aggregated, optimization exhibits an imbalance across action components, which hinders the modeling of fine-grained effects and affects action fidelity. We therefore propose DexAC-WM, which treats action conditioning as a structured process rather than global compression. DexAC preserves dimension-level semantics via action tokenization and aligns action signals with visual dynamics through local refinement and global modulation. To address the limited high-level semantic grounding in existing world models, we further introduce a semantic branch that provides rich object-scene priors, which enables world model to capture dynamic visual details while supporting high-DoF action-conditioned video prediction. Experiments on EgoDex and EgoVerse show that combining the semantic branch with DexAC significantly improves FID, FVD, and PCK, demonstrating gains in visual-temporal realism and action-following consistency. We further verify that DexAC extends to other backbones, showing the scalability of our structured action-conditioning design. These results suggest that scaling world models to high-DoF control requires both structured action modeling and semantic grounding.

Problem

• dexterous manipulation
• action conditioning
• world models
• high-DoF control
• action heterogeneity

Innovation

• structured action conditioning
• action tokenization
• semantic grounding
• high-DoF control
• world model