Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents

2026-06-22

AI Summary

This work addresses the challenge of failure detection in long-horizon robotic manipulation, where failures often emerge ambiguously and lack dense temporal annotations. The authors propose the first failure detection framework based on action-conditioned latent variables of a world model, trained exclusively with task-level success/failure labels. To enable consistent detection across diverse policies, they introduce Functional Conformal Prediction (FCP) for adaptive threshold calibration. Integrating vision-language-action policies, the method significantly outperforms existing approaches on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K simulation benchmarks and demonstrates effective real-world performance on ReacherX-200 and Franka robotic arms.

Abstract

Long-horizon tasks are common in real-world robotic deployments, yet failure detection for such tasks remains underexplored. Detecting failures in long-horizon robotic tasks is particularly challenging because failure onset is often ambiguous and dense temporal annotations are typically unavailable. We present Foresight, a failure detection framework that monitors manipulation trajectories using latent representations from an action-conditioned world model. Foresight is trained using only final task-level success or failure labels. By leveraging predictive world-model embeddings, our method provides a unified framework for failure detection across different policies. We further use functional conformal prediction (FCP) to calibrate detection thresholds adaptively. We evaluate Foresight with state-of-the-art vision-language-action policies in simulation on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K, compare it against state-of-the-artfailure detection methods, and validate it on real robots with three long-horizon tasks on a ReactorX-200 arm and one task on a Franka arm. Our results suggest that action-conditioned world-model embeddings provide a scalable representation for reliable failure monitoring in long-horizon manipulation.

Problem

• failure detection
• long-horizon robotic manipulation
• action-conditioned world model
• latent representations
• temporal annotations

Innovation

• action-conditioned world model
• failure detection
• long-horizon manipulation
• functional conformal prediction
• latent representation