ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning

2025-03-26

AI Summary

Autonomous agents operating in open environments face critical vulnerabilities to adversarial instructions and dynamic attacks, while existing defense mechanisms struggle to simultaneously ensure security and formal verifiability. To address this, we propose ShieldGuard—the first verifiable safety guard framework for agent action trajectories. Our contributions are threefold: (1) a novel probabilistic, action-level rule-circuit modeling method; (2) a formally verifiable safety guard agent that enforces strict adherence of action trajectories to explicit safety policies via logical reasoning; and (3) ShieldAgent-Bench, the first benchmark for evaluating agent guarding capabilities. Extensive experiments demonstrate state-of-the-art performance across ShieldAgent-Bench and three major public benchmarks, achieving an average accuracy improvement of 11.3% and a recall rate of 90.1%. Moreover, ShieldGuard reduces API calls by 64.7% and inference latency by 58.2%, significantly enhancing both efficiency and robustness.

Technology Category

• Multiagent Systems: Adversarial Agents
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Autonomous agents powered by foundation models have seen widespread adoption across various real-world applications. However, they remain highly vulnerable to malicious instructions and attacks, which can result in severe consequences such as privacy breaches and financial losses. More critically, existing guardrails for LLMs are not applicable due to the complex and dynamic nature of agents. To tackle these challenges, we propose ShieldAgent, the first guardrail agent designed to enforce explicit safety policy compliance for the action trajectory of other protected agents through logical reasoning. Specifically, ShieldAgent first constructs a safety policy model by extracting verifiable rules from policy documents and structuring them into a set of action-based probabilistic rule circuits. Given the action trajectory of the protected agent, ShieldAgent retrieves relevant rule circuits and generates a shielding plan, leveraging its comprehensive tool library and executable code for formal verification. In addition, given the lack of guardrail benchmarks for agents, we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent instructions and action trajectories, collected via SOTA attacks across 6 web environments and 7 risk categories. Experiments show that ShieldAgent achieves SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior methods by 11.3% on average with a high recall of 90.1%. Additionally, ShieldAgent reduces API queries by 64.7% and inference time by 58.2%, demonstrating its high precision and efficiency in safeguarding agents.

Problem

• Protecting autonomous agents from malicious attacks and instructions
• Ensuring safety policy compliance through verifiable logical reasoning
• Addressing lack of guardrail benchmarks for agent safety evaluation

Innovation

• Enforces safety via verifiable policy reasoning
• Constructs action-based probabilistic rule circuits
• Introduces ShieldAgent-Bench dataset for evaluation