Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective

2025-08-14

AI Summary

Current vision-driven robotic simulation benchmarks have advanced manipulation research but critically lack evaluation capabilities for sim-to-real transfer. Method: We introduce the first benchmark specifically designed for assessing general-purpose manipulation policies’ sim-to-real transferability—featuring a high-fidelity visual simulation environment, a controllable, incrementally complex task suite, a systematic domain perturbation protocol (e.g., lighting, material, motion blur), and novel cross-domain performance alignment metrics. Contribution/Results: Our framework is the first to jointly model task difficulty, perturbation type, and inter-domain performance gap, thereby explicitly exposing generalization bottlenecks of existing policies under realistic conditions. Experiments demonstrate the benchmark’s reproducibility, scalability, and diagnostic utility, establishing a standardized, rigorous testbed for evaluating and improving the robustness and transfer efficiency of general-purpose robotic policies.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Current vision-based robotics simulation benchmarks have significantly advanced robotic manipulation research. However, robotics is fundamentally a real-world problem, and evaluation for real-world applications has lagged behind in evaluating generalist policies. In this paper, we discuss challenges and desiderata in designing benchmarks for generalist robotic manipulation policies for the goal of sim-to-real policy transfer. We propose 1) utilizing high visual-fidelity simulation for improved sim-to-real transfer, 2) evaluating policies by systematically increasing task complexity and scenario perturbation to assess robustness, and 3) quantifying performance alignment between real-world performance and its simulation counterparts.

Problem

• Challenges in sim-to-real policy transfer benchmarks
• Evaluating robustness via task complexity and perturbations
• Quantifying sim-real performance alignment metrics

Innovation

• High visual-fidelity simulation for better transfer
• Systematic task complexity increase for robustness
• Quantify real-simulation performance alignment