RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

2026-04-10

AI Summary

Existing robotic simulation benchmarks struggle to effectively evaluate the generalization and robustness of task-agnostic policies due to significant overlap between training and evaluation domains. To address this limitation, this work proposes RoboLab-120, the first benchmark framework centered on a three-dimensional capability axis—visual, procedural, and relational—for assessing task-agnostic strategies. It features a high-fidelity, physically and visually realistic simulation environment that supports human–large language model collaboration in generating robot- and policy-agnostic task scenarios. A controllable perturbation system is introduced to systematically analyze policy sensitivity to external factors. Experiments reveal substantial performance gaps in cross-task generalization among state-of-the-art models, while providing fine-grained metrics and an extensible toolkit to enable systematic evaluation of true generalization capabilities.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Natural Language Processing: Safety and Robustness
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which external factors most strongly affect that behavior under controlled perturbations. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a physically realistic and photorealistic simulation. With this, we propose the RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational competency, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, indicating that high-fidelity simulation can serve as a proxy for analyzing performance and its dependence on external factors. Evaluation with RoboLab exposes significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies.

Problem

• simulation benchmarking
• task generalist policies
• generalization
• domain overlap
• robotic policy evaluation

Innovation

• high-fidelity simulation
• task generalist policies
• systematic perturbation analysis
• robot-agnostic benchmarking
• LLM-enabled task generation