Automating Safety Enhancement for LLM-based Agents with Synthetic Risk Scenarios

2025-05-23

AI Summary

To address security risks in LLM agents—such as privilege escalation, harmful outputs, and logical失控 during multi-turn interactions and tool invocation—this paper proposes AutoSafe, a fully automated safety-enhancement framework. Methodologically, AutoSafe introduces (i) an Open Threat Simulation (OTS) model that enables fully automated generation of high-risk scenarios without requiring real-world hazardous data; and (ii) an integrated pipeline combining synthetic user-behavior simulation, self-reflective reasoning, multi-stage safety-data distillation, and large-scale safety-oriented fine-tuning. Evaluated on standard safety benchmarks, AutoSafe achieves an average 45% improvement in safety metrics and enhances real-world task safety by 28.91%. Moreover, it significantly improves cross-scenario generalization and production-deployment reliability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

Large Language Model (LLM)-based agents are increasingly deployed in real-world applications such as"digital assistants, autonomous customer service, and decision-support systems", where their ability to"interact in multi-turn, tool-augmented environments"makes them indispensable. However, ensuring the safety of these agents remains a significant challenge due to the diverse and complex risks arising from dynamic user interactions, external tool usage, and the potential for unintended harmful behaviors. To address this critical issue, we propose AutoSafe, the first framework that systematically enhances agent safety through fully automated synthetic data generation. Concretely, 1) we introduce an open and extensible threat model, OTS, which formalizes how unsafe behaviors emerge from the interplay of user instructions, interaction contexts, and agent actions. This enables precise modeling of safety risks across diverse scenarios. 2) we develop a fully automated data generation pipeline that simulates unsafe user behaviors, applies self-reflective reasoning to generate safe responses, and constructs a large-scale, diverse, and high-quality safety training dataset-eliminating the need for hazardous real-world data collection. To evaluate the effectiveness of our framework, we design comprehensive experiments on both synthetic and real-world safety benchmarks. Results demonstrate that AutoSafe boosts safety scores by 45% on average and achieves a 28.91% improvement on real-world tasks, validating the generalization ability of our learned safety strategies. These results highlight the practical advancement and scalability of AutoSafe in building safer LLM-based agents for real-world deployment. We have released the project page at https://auto-safe.github.io/.

Problem

• Ensuring safety of LLM-based agents in dynamic interactions
• Automating synthetic risk scenario generation for safety training
• Improving agent safety without hazardous real-world data collection

Innovation

• Automated synthetic risk scenario generation
• Open extensible threat model OTS
• Self-reflective safe response generation