Active Real-World Factor-Based Evaluation for Generalist Robot Policies

2026-07-15

AI Summary

Evaluating general-purpose robotic policies in the real world is hindered by the combinatorial explosion of task factors, high testing costs, and insufficient coverage of existing test sets, making it difficult to comprehensively assess deployment reliability. This work formulates evaluation as a sequential experimental design problem and introduces the first active, factorized evaluation framework for real-world robot policies. By leveraging Bayesian optimization and probabilistic surrogate models, the approach adaptively selects test configurations with maximal information gain within a structured task factor space, efficiently characterizing policy performance under unseen conditions. Across three tasks and 2,331 real-world trials, the method reduces the number of required tests by 20–40% compared to random testing, significantly improving evaluation efficiency and coverage while systematically identifying failure regions.

Abstract

Generalist robot manipulation policies trained on large, diverse datasets have shown remarkable promise across a wide range of tasks. However, rigorously evaluating these policies remains a fundamental challenge. Real-world performance depends on a large combinatorial space of task factors including object poses and camera viewpoints, making full, exhaustive evaluation intractable. Additionally, real hardware evaluation is slow and resource-intensive, so current practice is to use narrow test suites that can miss critical failure modes and misrepresent true deployment readiness. We propose an active evaluation framework that addresses this challenge by treating policy evaluation as a sequential experimental design problem. Our approach fits a probabilistic surrogate model over a structured space of task factors and adaptively selects evaluation configurations to maximize information gain over the policy's performance distribution, allowing for sample-efficient characterization of policy behavior across unseen conditions and a systematic identification of failure-prone regions. We conduct 2331 real-world evaluations across 3 tasks with 3 factor variations and find that our approach typically saves the evaluator at least 20-40% of trials compared to typical random testing.

Problem

• robot policy evaluation
• real-world evaluation
• task factors
• failure modes
• deployment readiness

Innovation

• active evaluation
• generalist robot policies
• probabilistic surrogate model
• sequential experimental design
• sample-efficient evaluation