Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures

2025-05-01

AI Summary

Generative world models in vision-driven robotics exhibit overconfidence under out-of-distribution (OOD) conditions, causing safety filters to fail catastrophically. Method: We propose an implicit safety filter that jointly leverages representation and epistemic uncertainty. Specifically, we employ conformal prediction–calibrated epistemic uncertainty as an OOD hazard detection signal and perform Hamilton–Jacobi reachability analysis in the joint uncertainty–latent-state space to uniformly avoid both known and unknown hazards. The filter operates within an augmented state space and provides provably safe action filtering without assuming any specific policy architecture. Results: Evaluated on Franka Emika Panda robot simulations and real-world experiments, our method achieves 100% avoidance of OOD failures unseen during training while guaranteeing feasible safe actions at all times—significantly enhancing the robust safety of world-model-driven robotic systems.

Technology Category

• Intelligent Robots: State Estimation
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advances in generative world models have enabled classical safe control methods, such as Hamilton-Jacobi (HJ) reachability, to generalize to complex robotic systems operating directly from high-dimensional sensor observations. However, obtaining comprehensive coverage of all safety-critical scenarios during world model training is extremely challenging. As a result, latent safety filters built on top of these models may miss novel hazards and even fail to prevent known ones, overconfidently misclassifying risky out-of-distribution (OOD) situations as safe. To address this, we introduce an uncertainty-aware latent safety filter that proactively steers robots away from both known and unseen failures. Our key idea is to use the world model's epistemic uncertainty as a proxy for identifying unseen potential hazards. We propose a principled method to detect OOD world model predictions by calibrating an uncertainty threshold via conformal prediction. By performing reachability analysis in an augmented state space-spanning both the latent representation and the epistemic uncertainty-we synthesize a latent safety filter that can reliably safeguard arbitrary policies from both known and unseen safety hazards. In simulation and hardware experiments on vision-based control tasks with a Franka manipulator, we show that our uncertainty-aware safety filter preemptively detects potential unsafe scenarios and reliably proposes safe, in-distribution actions. Video results can be found on the project website at https://cmu-intentlab.github.io/UNISafe

Problem

• Detecting out-of-distribution failures in robotic control
• Improving safety filters with epistemic uncertainty awareness
• Preventing known and unseen hazards in vision-based tasks

Innovation

• Uses epistemic uncertainty to detect hazards
• Calibrates uncertainty threshold via conformal prediction
• Augments state space for reachability analysis