Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

2026-06-15

AI Summary

This work proposes the first embodied world model with natural language as a unified action interface across tasks, designed to predict physically plausible future visual trajectories from linguistic instructions. The approach employs a three-stage design: a dual-stream MMDiT architecture that fuses multimodal semantics with video latent variables, a large-scale embodied video–text corpus comprising 8.6 million samples, and a progressive curriculum training strategy that transitions from general to expert capabilities. It leverages a frozen Qwen2.5-VL encoder and operates within a video-VAE latent space. The model achieves top performance on EWMBench and DreamGen Bench, and significantly outperforms existing open-source models on WorldModelBench, PBench, and RoboTwin-IF, demonstrating strong zero-shot generalization and multi-view consistency. It enables three key applications: synthetic data generation, virtual evaluation, and language-guided planning.

Abstract

We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.

Problem

• embodied intelligence
• language-conditioned video generation
• world modeling
• future prediction
• unified action interface

Innovation

• language-conditioned video generation
• embodied world modeling
• double-stream MMDiT
• Embodied World Knowledge (EWK)
• progressive curriculum training