Conformal Safety Shielding for Imperfect-Perception Agents

2025-06-12

arXiv.org

AI Summary

Ensuring runtime safety for discrete autonomous agents—such as airport taxiway guidance systems—under imperfect perception, where high-dimensional observations induce state estimation errors. Method: We propose a safety-critical control framework integrating conformal prediction with runtime action shielding. Specifically, we introduce conformal prediction into safety shielding for the first time, designing an action pruning strategy grounded in a state estimate set; safety is formalized via a Markov decision process to guarantee local deterministic safety. Additionally, we derive, for the first time, the global safety probability bound for an ideal (perfect-perception) shielder. Results: Experiments on a real-world taxiway guidance system demonstrate that our approach significantly reduces boundary violation risk caused by perception errors, achieving reliable runtime safety at 95% confidence.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: State Estimation

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

We consider the problem of safe control in discrete autonomous agents that use learned components for imperfect perception (or more generally, state estimation) from high-dimensional observations. We propose a shield construction that provides run-time safety guarantees under perception errors by restricting the actions available to an agent, modeled as a Markov decision process, as a function of the state estimates. Our construction uses conformal prediction for the perception component, which guarantees that for each observation, the predicted set of estimates includes the actual state with a user-specified probability. The shield allows an action only if it is allowed for all the estimates in the predicted set, resulting in a local safety guarantee. We also articulate and prove a global safety property of existing shield constructions for perfect-perception agents bounding the probability of reaching unsafe states if the agent always chooses actions prescribed by the shield. We illustrate our approach with a case-study of an experimental autonomous system that guides airplanes on taxiways using high-dimensional perception DNNs.

Problem

• Safe control for agents with imperfect perception
• Run-time safety guarantees under perception errors
• Shielding actions based on conformal prediction sets

Innovation

• Conformal prediction for perception safety
• Run-time shield restricts unsafe actions
• Global safety proof for perfect-perception agents