GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

2026-07-02

AI Summary

This work addresses the reliance on costly and inefficient real-world deployment for robotic policy evaluation by systematically investigating world models as a viable alternative. The authors introduce WMBench, a benchmark that integrates real teleoperated and policy-executed trajectories to enable controlled comparisons across model architectures, action representations, and evaluation metrics. Centered on long-horizon action consistency as a core evaluation criterion, the study evaluates seven video world models with four action encodings, incorporating memory mechanisms, optimized action representations, and evaluation-oriented post-training strategies on large-scale simulated and real-world data. Analysis of 324,000 policy execution pairs reveals the critical impact of pretraining data balance and architectural design on evaluation reliability, substantially improving alignment between world model predictions and real-world behavior. The authors release the GigaWorld-1 model, dataset, and full toolchain to support future research.

Abstract

Evaluating embodied robot foundation models remains a critical bottleneck; unlike large language models efficiently assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts limited by hardware and human supervision, which has driven interest in world models as surrogate policy evaluators, yet the key properties that make a world model reliable for policy assessment remain poorly understood. This work presents a systematic study of world models for robotic policy evaluation and introduces WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks to enable controlled comparisons across model families, action encodings, rollout horizons, and evaluation metrics. Using WMBench, we analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, further enriching our analysis with large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and a training videos spanning more than 12,000 hours. Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism; pretraining gains stem not only from data scale but from balancing general world knowledge with robot-specific controllability; and architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior. Drawing on these results, we derive a practical design roadmap and realize it in \textit{GigaWorld-1}, a world model specially optimized for policy evaluation, and we fully release our code, models, datasets, and toolkits to advance scalable evaluation research for embodied foundation models.

Problem

• world models
• robot policy evaluation
• embodied foundation models
• policy assessment
• surrogate evaluation

Innovation

• world models
• policy evaluation
• robotic foundation models
• WMBench
• action-faithful rollout