What You Don't Know Can Hurt You: How Well do Latent Safety Filters Understand Partially Observable Safety Constraints?

2025-10-07

AI Summary

In partially observable environments relying solely on RGB observations, existing latent-space safety filters struggle to capture unseen safety constraints, leading to myopic safety behaviors. Method: We propose a mutual information-based metric to quantify observation completeness; design a multimodal supervised training strategy to enhance the latent space’s capacity to model implicit safety states; and integrate world-model learning of latent dynamics with Hamilton–Jacobi reachability analysis and a classification-based mechanism to construct a robust safety controller. Contribution: This work is the first to systematically expose the fundamental limitations of latent-space safety modeling under RGB-only observation. Evaluated in simulation and on a Franka Research 3 robotic arm, our approach effectively prevents latent hazards—such as overheating of a wax pot—demonstrating substantial improvements in both robustness and generalization of safety-critical control.

Technology Category

• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Safe control techniques, such as Hamilton-Jacobi reachability, provide principled methods for synthesizing safety-preserving robot policies but typically assume hand-designed state spaces and full observability. Recent work has relaxed these assumptions via latent-space safe control, where state representations and dynamics are learned jointly through world models that reconstruct future high-dimensional observations (e.g., RGB images) from current observations and actions. This enables safety constraints that are difficult to specify analytically (e.g., spilling) to be framed as classification problems in latent space, allowing controllers to operate directly from raw observations. However, these methods assume that safety-critical features are observable in the learned latent state. We ask: when are latent state spaces sufficient for safe control? To study this, we examine temperature-based failures, comparable to overheating in cooking or manufacturing tasks, and find that RGB-only observations can produce myopic safety behaviors, e.g., avoiding seeing failure states rather than preventing failure itself. To predict such behaviors, we introduce a mutual information-based measure that identifies when observations fail to capture safety-relevant features. Finally, we propose a multimodal-supervised training strategy that shapes the latent state with additional sensory inputs during training, but requires no extra modalities at deployment, and validate our approach in simulation and on hardware with a Franka Research 3 manipulator preventing a pot of wax from overheating.

Problem

• Evaluating latent safety filters' understanding of partially observable constraints
• Identifying when RGB observations produce myopic safety behaviors
• Proposing multimodal training to capture safety-critical latent features

Innovation

• Latent-space safe control using world models
• Mutual information measure for safety feature observability
• Multimodal-supervised training without extra deployment modalities