SAFE: Multitask Failure Detection for Vision-Language-Action Models

2025-06-11

AI Summary

Visual-language-action (VLA) models often fail to generalize fault detection to unseen tasks and novel environments. Method: This paper introduces the first multi-task fault detection task for VLAs. We propose a lightweight, single-scalar detector built upon intermediate VLA features, trained via supervised learning and calibrated using conformal prediction—enabling zero-shot cross-task transfer without task labels. The approach is architecture-agnostic, compatible with OpenVLA, π₀, and other mainstream VLAs, and deployable in both simulation and real-world robotic systems. Contribution/Results: We empirically reveal that VLA feature spaces encode task-agnostic success/failure semantics; and establish the first generalizable fault detection framework for universal robotic policies. Experiments across multiple models and environments demonstrate significant improvements over baselines: +12.3% average AUC and 1.8-second average early warning latency—achieving a new state-of-the-art trade-off between detection accuracy and response speed.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

While vision-language-action models (VLAs) have shown promising robotic behaviors across a diverse set of manipulation tasks, they achieve limited success rates when deployed on novel tasks out-of-the-box. To allow these policies to safely interact with their environments, we need a failure detector that gives a timely alert such that the robot can stop, backtrack, or ask for help. However, existing failure detectors are trained and tested only on one or a few specific tasks, while VLAs require the detector to generalize and detect failures also in unseen tasks and novel environments. In this paper, we introduce the multitask failure detection problem and propose SAFE, a failure detector for generalist robot policies such as VLAs. We analyze the VLA feature space and find that VLAs have sufficient high-level knowledge about task success and failure, which is generic across different tasks. Based on this insight, we design SAFE to learn from VLA internal features and predict a single scalar indicating the likelihood of task failure. SAFE is trained on both successful and failed rollouts, and is evaluated on unseen tasks. SAFE is compatible with different policy architectures. We test it on OpenVLA, $pi_0$, and $pi_0$-FAST in both simulated and real-world environments extensively. We compare SAFE with diverse baselines and show that SAFE achieves state-of-the-art failure detection performance and the best trade-off between accuracy and detection time using conformal prediction. More qualitative results can be found at https://vla-safe.github.io/.

Problem

• Detect failures in vision-language-action models for novel tasks
• Generalize failure detection across unseen tasks and environments
• Provide timely alerts for safe robot interaction and recovery

Innovation

• Uses VLA internal features for failure detection
• Generalizes across unseen tasks and environments
• Achieves best accuracy and detection time trade-off