AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

2024-10-11

arXiv.org

AI Summary

This work addresses a novel security threat to LLM-based agents—harmful behavior induced by jailbreaking attacks during tool invocation and multi-step task execution. We introduce AgentHarm, the first agent-oriented harmfulness evaluation benchmark, comprising 11 harm categories, 110 malicious agent tasks, and 440 semantically augmented instances. It enables systematic assessment of multi-step behavioral safety under jailbreaking. Methodologically, we propose a dual-dimension scoring metric—“rejection rate” and “functional coherence”—to jointly measure safety compliance (i.e., refusal capability) and post-jailbreak task execution reasonableness. Our approach integrates human-crafted task generation with semantic augmentation, cross-scenario adaptation of generic jailbreaking templates, and a multi-step behavioral consistency evaluation framework; all data are open-sourced on Hugging Face. Experiments reveal that mainstream LLM agents exhibit high obedience to malicious instructions even without jailbreaking, and generic jailbreaking templates transfer effectively across scenarios, triggering coherent harmful behaviors.

Technology Category

• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as simple chatbots. Meanwhile, LLM agents -- which use external tools and can execute multi-stage tasks -- may pose a greater risk if misused, but their robustness remains underexplored. To facilitate research on LLM agent misuse, we propose a new benchmark called AgentHarm. The benchmark includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. In addition to measuring whether models refuse harmful agentic requests, scoring well on AgentHarm requires jailbroken agents to maintain their capabilities following an attack to complete a multi-step task. We evaluate a range of leading LLMs, and find (1) leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking, (2) simple universal jailbreak templates can be adapted to effectively jailbreak agents, and (3) these jailbreaks enable coherent and malicious multi-step agent behavior and retain model capabilities. To enable simple and reliable evaluation of attacks and defenses for LLM-based agents, we publicly release AgentHarm at https://huggingface.co/datasets/ai-safety-institute/AgentHarm.

Problem

• Measure LLM agent harmfulness via diverse malicious tasks
• Assess jailbreak robustness in multi-step agent scenarios
• Evaluate compliance of leading LLMs to harmful requests

Innovation

• Proposes AgentHarm benchmark for LLM agent misuse
• Includes 110 malicious tasks across 11 harm categories
• Evaluates jailbreak robustness in multi-step agent tasks