Agent-SafetyBench: Evaluating the Safety of LLM Agents

2024-12-19

arXiv.org

AI Summary

Existing security evaluations for LLM-based agents lack systematic benchmarks to address novel safety challenges arising from tool invocation and environment interaction. Method: We introduce SafeAgent-Bench—the first dedicated security evaluation benchmark for LLM agents—comprising 349 interactive environments and 2,000 test cases. It systematically assesses eight safety risk categories and ten failure modes, incorporating multi-scenario red-teaming, structured risk taxonomy, standardized safety scoring, and an open-source, extensible platform. Contribution/Results: We formally define and quantify agent-specific safety dimensions, identifying “robustness deficiency” and “risk awareness deficiency” as fundamental flaws; empirical analysis shows defensive prompting yields limited mitigation. Evaluating 16 state-of-the-art agents, the highest safety score achieved is only 58.7%. SafeAgent-Bench is publicly released to advance standardization and methodology in agent safety evaluation.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Adversarial Agents

Application Category

• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

As large language models (LLMs) are increasingly deployed as agents, their integration into interactive environments and tool use introduce new safety challenges beyond those associated with the models themselves. However, the absence of comprehensive benchmarks for evaluating agent safety presents a significant barrier to effective assessment and further improvement. In this paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions. Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%. This highlights significant safety challenges in LLM agents and underscores the considerable need for improvement. Through quantitative analysis, we identify critical failure modes and summarize two fundamental safety detects in current LLM agents: lack of robustness and lack of risk awareness. Furthermore, our findings suggest that reliance on defense prompts alone is insufficient to address these safety issues, emphasizing the need for more advanced and robust strategies. We release Agent-SafetyBench at url{https://github.com/thu-coai/Agent-SafetyBench} to facilitate further research and innovation in agent safety evaluation and improvement.

Problem

• Evaluating safety risks in LLM agents
• Identifying lack of robustness and risk awareness
• Assessing insufficiency of defense prompts

Innovation

• Comprehensive benchmark for LLM agent safety
• Evaluates 8 safety risks and 10 failure modes
• Highlights need for advanced safety strategies