Coverage Aware Active Evaluation for Failure Discovery with Paired Systems

2026-08-13

AI Summary

This study addresses the challenges of discovering rare heterogeneous failures in autonomous systems under limited budgets and the poor transferability of surrogate signals. We propose an adaptive failure discovery framework that mitigates risk prediction bias through controlled variable residual modeling and optimizes scenario selection via support-aware mutual information maximization, thereby balancing prediction accuracy with pattern coverage diversity. Experimental results demonstrate that this approach doubles the failure detection rate compared to baselines in autonomous driving tasks. It efficiently identifies genuine critical failures while effectively covering overlooked long-tail scenarios, significantly enhancing both testing efficiency and system safety.

Abstract

Autonomous systems can fail in rare and heterogeneous ways, making real-world failure discovery difficult under limited testing budgets. Although cheaper proxies such as simulators, lower-fidelity systems, or related policies can be sampled extensively to find failures, proxy failures often do not transfer to the real world due to sim-to-real and system-to-system gaps. The key challenge is therefore to effectively leverage proxy system information for accurate prediction of severe target system failures. We propose an adaptive failure discovery method that combines proxy evaluations with limited target system results to guide scenario selection for target system testing. Our method learns a local predictor of target risk by correcting proxy failure signals using control-variate-inspired residual modeling. To find failures that are both likely and diverse, we combine this predictor with a support-aware mutual-information objective that favors realistic, well-supported regions while expanding coverage across failure modes. Across autonomous driving, manipulation, and quadruped velocity-tracking tasks, our method discovers up to 2$\times$ as many failures as random sampling and active-learning baselines, including severe and diverse failures missed by competing methods.

Problem

• Failure Discovery
• Autonomous Systems
• Proxy-to-Real Gap
• Active Evaluation
• Limited Testing Budget

Innovation

• Active Evaluation
• Control-Variate Residual Modeling
• Support-Aware Mutual Information
• Failure Discovery
• Coverage Awareness