Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

2024-10-03

arXiv.org

AI Summary

Existing evaluations of LLM-based agents lack systematic assessment of security vulnerabilities arising from tool invocation, memory retrieval, and system prompting in real-world tasks. Method: We propose ASB—the first full-stack security benchmark for LLM agents—covering 10 realistic scenarios, 10 mainstream agent architectures, and 400+ tools. ASB formally defines full-stack security threats, systematically models 27 attack-defense techniques, and introduces a novel Plan-of-Thought backdoor attack. It further devises a utility-security trade-off metric and establishes a multi-dimensional quantitative evaluation framework. Results: Extensive experiments across 13 prominent LLM backbones reveal an average attack success rate of 84.30%; existing defenses consistently fail. These findings underscore the urgency of agent security research and establish ASB as a foundational benchmark for rigorous, holistic security evaluation of LLM agents.

Technology Category

• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https://github.com/agiresearch/ASB.

Problem

• Formalizing and benchmarking attacks on LLM-based agents
• Evaluating defenses against vulnerabilities in LLM-based agents
• Introducing a new metric for balancing utility and security

Innovation

• ASB framework formalizes LLM agent security.
• Benchmarks 27 attack/defense methods across scenarios.
• Introduces new metric for utility-security balance.