On Safety Risks in Experience-Driven Self-Evolving Agents

2026-04-18

AI Summary

While experience-driven self-evolving agents enhance autonomy, their reliance on self-sampled experience introduces underappreciated safety risks. This work systematically uncovers a critical safety degradation issue rooted in the “execution-oriented” nature of such mechanisms, demonstrating that even benign task experiences can compromise safety in high-risk scenarios. Through empirical studies in both web and embodied environments—employing experience replay analysis, task categorization, and modeling of refusal behaviors—we find that incorporating refusal experiences partially mitigates these risks but often leads to excessive refusals. Our findings reveal a fundamental trade-off between safety and utility inherent in current self-evolution paradigms, exposing their intrinsic limitations in balancing reliable performance with robust safety guarantees.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Experience-driven self-evolution has emerged as a promising paradigm for improving the autonomy of large language model agents, yet its reliance on self-curated experience introduces underexplored safety risks. In this study, we investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments. Notably, experience gathered solely from benign tasks can still compromise safety in high-risk scenarios. Further analysis attributes this degradation to the execution-oriented nature of accumulated experience, which reinforces agents' tendency to act rather than refuse. In more realistic settings where agents encounter both benign and harmful tasks, refusal-related experience mitigates safety decline but induces over-refusal, revealing a fundamental safety-utility trade-off. Overall, our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation.

Problem

• self-evolving agents
• safety risks
• experience-driven learning
• refusal behavior
• safety-utility trade-off

Innovation

• self-evolving agents
• experience-driven learning
• safety risks
• refusal behavior
• safety-utility trade-off