Betting for Sim-to-Real Performance Evaluation

2026-04-26

AI Summary

This work proposes a novel “betting”-based methodology for efficiently and accurately evaluating robotic performance in real-world settings where physical experimentation is constrained. By introducing betting theory into sim-to-real performance assessment—a first in the field—the approach constructs an estimator theoretically superior to Monte Carlo estimation. The method integrates control variate approximation, cross-fidelity simulation, and statistical decision rules to enable practical deployment. Experimental results demonstrate its efficacy on synthetic data and simulated environments, and it successfully infers real-world robotic grasping accuracy with significantly fewer physical trials while improving evaluation precision.

Technology Category

• Intelligent Robots: State Estimation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

This paper studies the problem of robot performance evaluation, focusing on how to obtain accurate and efficient estimates of real-world behavior under severe constraints on physical experimentation. Such estimates are essential for benchmarking algorithms, comparing design alternatives, validating controllers, and supporting certification or regulatory decision-making, yet real-world testing with physical robots is often expensive, time-consuming, and safety-limited. To mitigate the scarcity of real-world trials, sim-to-real methodologies are commonly employed, using low-cost simulators to inform, supplement, or prioritize physical experiments. Departing from (and complementary to) existing approaches in variance reduction (e.g., importance-sampling variants) or bias-correction (e.g., through prediction-powered inference or learned control variates), we examine this performance-evaluation problem through the lens of betting. We establish theoretical conditions under which a betting mechanism can yield accurate and efficient estimates (provably outperforming the Monte Carlo estimator) and we characterize how such bets should be constructed. We further develop theoretically grounded yet practically implementable approximations of the ideal bet, and we provide concrete decision rules that diagnose when these approximate betting strategies are working as intended. We demonstrate the effectiveness of the proposed methods using both synthetic examples and cross-fidelity computational simulators. Notably, we also showcase an illustrative case in which a group of synthetic distributions are used to infer the real-world pick-and-place accuracy of a robotic manipulator, a seemingly unconventional sim-to-real transfer that becomes natural and feasible under the proposed betting perspective. Programs for reproducing empirical results are available at https://github.com/ISUSAIL/Bet4Sim2Real.

Problem

• sim-to-real
• performance evaluation
• robotics
• physical experimentation constraints
• real-world behavior estimation

Innovation

• betting
• sim-to-real
• performance evaluation
• Monte Carlo estimation
• robotics