OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences

2026-05-18

AI Summary

This work addresses a critical vulnerability in self-evolving large language model (LLM) agents, which often form harmful overgeneralizations from locally plausible yet non-transferable experiences during reflection, leading to task failure. The authors propose a novel low-privilege black-box attack that, for the first time, leverages the agent’s own clean but misleading self-generated experiences as attack vectors. Without requiring control over system prompts or memory banks, the method constructs semantically coherent yet consequential adversarial experiences through adversarial example generation, reflection mechanism analysis, and memory consolidation modeling, thereby inducing the agent to encode high-priority erroneous rules during memory integration. Evaluated across three task domains, the approach achieves over 50% attack success rate against GPT-4o, significantly outperforming existing methods and effectively evading LLM-based auditing defenses.

Abstract

Memory-augmented large language model (LLM) agents use iterative reflection and self-evolution to solve complex tasks, but these mechanisms introduce security risks. Existing agentic memory attacks require privileged access or explicit malicious content, making them detectable by advanced safety filters. This leaves a subtler attack surface underexplored: whether adversaries can induce agent to generate experiences that appear locally correct and semantically plausible yet induce harmful generalization during reflection. We find that reflective agents are vulnerable to such clean experiences, especially when paired with severe but plausible hypothetical consequences. Based on this observation, we introduce Obsessive Experience Poisoning (OEP), a low-privilege black-box attack requiring no direct control over the system prompt or memory database. OEP constructs adversarial clean edge-cases that combine locally correct solutions, non-transferable methods, and severe consequences, biasing reflection toward risk-averse rule formation. During memory consolidation, agents may over-trust self-generated reflections and distill localized experiences into high-priority but over-generalized rules, causing downstream failures. Evaluations across three domains show that OEP achieves ASR above 50\% with GPT-4o agents, and outperforms existing attacks under LLM auditing defense.

Problem

• LLM agents
• experience poisoning
• self-evolution
• reflection
• adversarial attack

Innovation

• Obsessive Experience Poisoning
• self-evolving LLM agents
• adversarial clean experiences
• memory consolidation
• non-transferable methods