VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models

2026-06-19

AI Summary

This work addresses the unpredictable behavior of vision-language-action (VLA) models in out-of-distribution scenarios by proposing a lightweight, general-purpose runtime task failure detection framework. Without requiring failure examples or costly sampling, the method innovatively combines two unsupervised mechanisms: Mahalanobis distance computed on last-layer features to identify out-of-distribution states, and temporal inconsistency detection leveraging action block overlap in receding-horizon control. To comprehensively evaluate detection performance, the authors introduce a new metric, AUCPDT, which jointly accounts for precision, recall, and timeliness. Experiments demonstrate that the approach reliably detects diverse task failures early in both real-world and simulated environments, outperforming more computationally expensive baselines.

Abstract

Vision-language-action models (VLAs) achieve state-of-the-art performance on many robotic manipulation tasks, yet they can still behave unpredictably in out-of-distribution scenarios. Runtime failure detection is therefore essential for the safe real-world deployment of VLAs. However, existing task failure detectors require computationally expensive action sampling, are based on architectural assumptions that limit their applicability to VLAs, or need access to failure rollouts. We propose VLA-FAIL, a lightweight and broadly applicable failure detection framework for VLAs that combines two novel failure detectors with minimal overhead, without requiring failure data. The first, last-layer Mahalanobis distance (LLMD), detects out-of-distribution states by measuring token-wise deviations in last-layer features relative to the training data. The second, action chunk consistency (ACC), exploits the temporal overlap induced by receding-horizon control and detects failures when consecutive action chunks become inconsistent. To capture the trade-off between detection accuracy and detection latency, we introduce AUCPDT, a threshold-independent metric that jointly evaluates precision, recall, and detection time. Through extensive real-world and simulation experiments, we demonstrate that LLMD and ACC capture complementary failure modes whose combination enables reliable and early failure detection across diverse tasks, frequently outperforming significantly more expensive baseline methods.

Problem

• Vision-Language-Action Models
• Task Failure Detection
• Out-of-Distribution
• Runtime Monitoring
• Robotic Manipulation

Innovation

• VLA-FAIL
• failure detection
• Mahalanobis distance
• action chunk consistency
• AUCPDT