World Guidance: World Modeling in Condition Space for Action Generation

2026-02-25

AI Summary

Existing vision-language-action (VLA) models struggle to balance efficient prediction with the preservation of fine-grained details when leveraging future observations to guide action generation, limiting both action accuracy and generalization. This work proposes WoG, a novel framework that performs world modeling in a conditional latent space: it compresses future observations into compact conditions and jointly predicts these conditions alongside actions, enabling efficient yet detail-preserving inference. By integrating conditional space mapping, future observation compression, and end-to-end joint modeling of vision, language, and action, WoG significantly enhances action generation quality. Experiments demonstrate that WoG outperforms existing future-prediction-based approaches in both simulation and real-world environments, achieving finer-grained action control and stronger cross-scenario generalization.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models. However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation. To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline. The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference. We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos. Extensive experiments across both simulation and real-world environments validate that our method significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/

Problem

• World Modeling
• Action Generation
• Future Observation
• Vision-Language-Action
• Condition Space

Innovation

• World Modeling
• Condition Space
• Action Generation
• Vision-Language-Action
• Future Prediction