RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

2026-07-05

AI Summary

Existing benchmarks for evaluating general-purpose robotic manipulation policies often suffer from simplistic tasks, limited evaluation dimensions, and a disconnect between simulation and real-world performance, hindering comprehensive assessment of policy capabilities. To address these limitations, this work introduces RoboDojo—the first standardized evaluation framework that enables coordinated simulation-to-reality benchmarking. RoboDojo encompasses 42 simulated and 18 real-world tasks, systematically evaluating policies across multiple dimensions including generalization, memory, precision, long-horizon execution, and open-vocabulary understanding. Built upon Isaac Sim for heterogeneous parallel simulation, the framework integrates the RoboDojo-RealEval physical evaluation system, the XPolicyLab unified policy interface, and cloud-based automated reset technology. It currently supports 30 diverse policies and features a public leaderboard, offering the community a reproducible and extensible infrastructure for robotic policy evaluation.

Abstract

Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.

Problem

• robot manipulation
• generalist policies
• sim-to-real benchmark
• comprehensive evaluation
• real-world deployment

Innovation

• sim-and-real benchmark
• generalist robot manipulation
• heterogeneous parallel simulation
• reproducible real-world evaluation
• XPolicyLab