DynaWM: A Base-VLA-Guided World Foundation Model for Moving-Object Manipulation

2026-07-01

AI Summary

This work addresses the limited performance of existing vision-language-action (VLA) models in manipulating dynamically moving objects, a challenge exacerbated by end-to-end fine-tuning that often degrades pretrained capabilities. To overcome this, we propose DynaWM—a world model guided by a foundation VLA—that decouples the VLA from the world model and integrates multi-view visual history, proprioceptive states, and primitive action blocks to generate motion-aware action trajectories. We introduce the first DynaGrasp-32 benchmark and DynaGrasp-1600 dataset, and implement conditional trajectory generation using a Mamba-3 action encoder, V-JEPA 2.1 visual encoder, and a flow-matching DiT architecture. Experiments demonstrate that DynaWM improves success rates by up to 45.31% and 44.06% over fine-tuned and coarsely adapted VLAs, respectively, with ablation studies confirming that the visual encoder boosts success by 27.50% and that action conditioning is indispensable.

Abstract

Although vision-language-action (VLA) models have received widespread attention, many challenges remain in manipulating dynamic moving objects. In most existing approaches, end-to-end forward or inverse dynamics models, i.e., world models, are incorporated into high-performance base VLA architectures, which may degrade the performance of well-pretrained base VLA models due to inappropriate fine-tuning. In this paper, we propose DynaWM, a base-VLA-guided world foundation model that adapts to a wide variety of fine-tuned and coarse-tuned base-VLA checkpoints for moving-object manipulation. DynaWM uses a Mamba-3-based action encoder to encode the base action chunk produced by the base VLA into an action-conditioning representation, a V-JEPA 2.1 vision encoder to extract features from multi-view observation history, and a proprioceptive state encoder to encode robotic-arm proprioceptive states. These feature representations jointly condition a flow-matching DiT to regenerate motion-aware action trajectories for moving-object manipulation. For systematic evaluation, we construct the DynaGrasp-32 benchmark, covering six categories of moving-object manipulation tasks, including velocity variation, trajectory variation, and multi-object manipulation, as well as the DynaGrasp-1600 dataset, which consists of 32 scenarios, 1,600 demonstration trajectories, and approximately 1.53M images. For fine-tuned base-VLA checkpoints, DynaWM achieves percentage improvements of 7.19, 45.31, 1.88, and 10.94 over SmolVLA, X-VLA, π0, and π0.5, respectively. For coarse-tuned base-VLA checkpoints, performance increases by 35.13, 44.06, 35.69, and 26.13 percentage, respectively. Ablation experiments show that visual encoding enhances success by 27.50%, while reducing success by 45.44% if action conditioning is removed.

Problem

• moving-object manipulation
• vision-language-action models
• world models
• dynamic objects
• VLA fine-tuning

Innovation

• world foundation model
• moving-object manipulation
• base-VLA-guided
• flow-matching DiT
• Mamba-3 action encoder