AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models

2025-05-29

AI Summary

Large foundation models for embodied agents suffer from susceptibility to misuse and lack of safety alignment during post-training. Method: This paper proposes an Abstract Action Chain (AAC)-based data synthesis framework that models high-level behavioral sequences and instantiates tool invocations in simulation environments, enabling semantic decoupling and co-generation of malicious and benign instructions while supporting controllable, multi-step balancing of harmfulness and usefulness for the first time. Contribution/Results: Evaluated on the AgentHarm benchmark, our approach elevates safety rates of three open-source models from 35.8% to 79.5%, without compromising—indeed often improving—task completion performance. It significantly outperforms prompt-engineering baselines. We publicly release a high-quality safety-aligned dataset and implementation code.

Technology Category

• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The acquisition of agentic capabilities has transformed LLMs from"knowledge providers"to"action executors", a trend that while expanding LLMs' capability boundaries, significantly increases their susceptibility to malicious use. Previous work has shown that current LLM-based agents execute numerous malicious tasks even without being attacked, indicating a deficiency in agentic use safety alignment during the post-training phase. To address this gap, we propose AgentAlign, a novel framework that leverages abstract behavior chains as a medium for safety alignment data synthesis. By instantiating these behavior chains in simulated environments with diverse tool instances, our framework enables the generation of highly authentic and executable instructions while capturing complex multi-step dynamics. The framework further ensures model utility by proportionally synthesizing benign instructions through non-malicious interpretations of behavior chains, precisely calibrating the boundary between helpfulness and harmlessness. Evaluation results on AgentHarm demonstrate that fine-tuning three families of open-source models using our method substantially improves their safety (35.8% to 79.5% improvement) while minimally impacting or even positively enhancing their helpfulness, outperforming various prompting methods. The dataset and code have both been open-sourced.

Problem

• Addressing safety risks in agentic LLMs from malicious use
• Improving post-training safety alignment for agentic LLM behaviors
• Balancing model safety and utility in multi-step agentic tasks

Innovation

• Uses abstract behavior chains for safety alignment
• Simulates diverse tool instances for authentic instructions
• Balances benign and harmful instruction synthesis