AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning

2026-04-09

AI Summary

This study addresses the challenge of industrial anomaly detection, where real anomalous samples are scarce and existing synthesis methods struggle to generate semantically realistic and diverse anomalies. To overcome this limitation, the work introduces an embodied agent paradigm for anomaly synthesis, proposing a novel anomaly synthesis agent endowed with self-reflection, knowledge retrieval, and iterative optimization capabilities. The agent leverages structured trajectories and a triple-reward mechanism to enable closed-loop refinement, augmented by tool-enhanced reinforcement learning within a two-stage training framework. Evaluated on MVTec-AD, the method achieves an Inception Score (IS) of 2.10 and Improved Consistency Loss (IC-L) of 0.33, attains a ResNet34 classification accuracy of 57.0%, and yields UNet image- and pixel-level average precisions of 99.3% and 74.2%, respectively—significantly outperforming current zero-shot state-of-the-art approaches.

Abstract

Industrial anomaly generation is a crucial method for alleviating the data scarcity problem in anomaly detection tasks. Most existing anomaly synthesis methods rely on single-step generation mechanisms, lacking complex reasoning and iterative optimization capabilities, making it difficult to generate anomaly samples with high semantic realism. We propose AnomalyAgent, an anomaly synthesis agent with self-reflection, knowledge retrieval, and iterative refinement capabilities, aiming to generate realistic and diverse anomalies. Specifically, AnomalyAgent is equipped with five tools: Prompt Generation (PG), Image Generation (IG), Quality Evaluation (QE), Knowledge Retrieval (KR), and Mask Generation (MG), enabling closed-loop optimization. To improve decision-making and self-reflection, we construct structured trajectories from real anomaly images and design a two-stage training framework: supervised fine-tuning followed by reinforcement learning. This process is driven by a three-part reward mechanism: (1) task rewards to supervise the quality and location rationality of generated anomalies; (2) reflection rewards to train the model's ability to improve anomaly synthesis prompt; (3) behavioral rewards to ensure adherence to the trajectory. On the MVTec-AD dataset, AnomalyAgent achieves IS/IC-L of 2.10/0.33 for anomaly generation, 57.0% classification accuracy using ResNet34, and 99.3%/74.2% AP at the image/pixel level using a simple UNet, surpassing all zero-shot SOTA methods. The code and data will be made publicly available.

Problem

• industrial anomaly generation
• data scarcity
• anomaly detection
• semantic realism
• anomaly synthesis

Innovation

• tool-augmented reinforcement learning
• agentic anomaly synthesis
• self-reflection
• iterative refinement
• structured trajectory