AnySafe: Adapting Latent Safety Filters at Runtime via Safety Constraint Parameterization in the Latent Space

2025-09-23

AI Summary

Existing latent-space safety filters—e.g., those based on Hamilton–Jacobi reachability analysis—rely on predefined, static safety constraints, limiting adaptability to dynamic task scenarios. This paper proposes a runtime-adaptive latent-space safety filter that parameterizes safety constraints as visual images and enables real-time, conditional encoding-based safety modulation within the world model’s latent space. It is the first method to support zero-shot adaptation to previously unseen safety constraint images and introduces conformal calibration of similarity metrics to ensure controllable safety during constraint approximation. The approach integrates Hamilton–Jacobi reachability analysis, latent-space modeling, and imagination-based offline training—requiring no online learning. Evaluated on both Franka Emika robot simulations and real hardware, the method achieves millisecond-level visual safety constraint response without compromising task performance.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent works have shown that foundational safe control methods, such as Hamilton-Jacobi (HJ) reachability analysis, can be applied in the latent space of world models. While this enables the synthesis of latent safety filters for hard-to-model vision-based tasks, they assume that the safety constraint is known a priori and remains fixed during deployment, limiting the safety filter's adaptability across scenarios. To address this, we propose constraint-parameterized latent safety filters that can adapt to user-specified safety constraints at runtime. Our key idea is to define safety constraints by conditioning on an encoding of an image that represents a constraint, using a latent-space similarity measure. The notion of similarity to failure is aligned in a principled way through conformal calibration, which controls how closely the system may approach the constraint representation. The parameterized safety filter is trained entirely within the world model's imagination, treating any image seen by the model as a potential test-time constraint, thereby enabling runtime adaptation to arbitrary safety constraints. In simulation and hardware experiments on vision-based control tasks with a Franka manipulator, we show that our method adapts at runtime by conditioning on the encoding of user-specified constraint images, without sacrificing performance. Video results can be found on https://any-safe.github.io

Problem

• Adapting safety filters to user-specified constraints at runtime
• Enabling runtime adaptation to arbitrary safety constraints dynamically
• Overcoming fixed safety constraints that limit adaptability across scenarios

Innovation

• Adapts safety filters at runtime via constraint parameterization
• Uses latent-space similarity measure for constraint encoding
• Trains entirely within world model's imagination for adaptation