ASBench: Image Anomalies Synthesis Benchmark for Anomaly Detection

2025-10-09

AI Summary

In industrial visual inspection, anomalous samples are scarce and costly to annotate, while existing anomaly synthesis methods lack systematic evaluation—particularly regarding cross-dataset generalization, sensitivity to synthetic ratio, correlation between synthesis quality and detection performance, and efficacy of hybrid strategies. To address this gap, we propose ASBench, the first comprehensive benchmark for image-level anomaly synthesis. ASBench decouples evaluation into four dimensions: cross-domain generalization, ratio sensitivity, metric–performance correlation, and hybrid strategy effectiveness. Through large-scale experiments across diverse models and datasets, we empirically expose critical limitations of current approaches. ASBench establishes the first standardized, reproducible evaluation framework for anomaly synthesis, providing quantitative analysis tools and actionable optimization guidelines. It serves as a foundational benchmark to advance principled research and development in anomaly synthesis for industrial inspection.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Anomaly detection plays a pivotal role in manufacturing quality control, yet its application is constrained by limited abnormal samples and high manual annotation costs. While anomaly synthesis offers a promising solution, existing studies predominantly treat anomaly synthesis as an auxiliary component within anomaly detection frameworks, lacking systematic evaluation of anomaly synthesis algorithms. Current research also overlook crucial factors specific to anomaly synthesis, such as decoupling its impact from detection, quantitative analysis of synthetic data and adaptability across different scenarios. To address these limitations, we propose ASBench, the first comprehensive benchmarking framework dedicated to evaluating anomaly synthesis methods. Our framework introduces four critical evaluation dimensions: (i) the generalization performance across different datasets and pipelines (ii) the ratio of synthetic to real data (iii) the correlation between intrinsic metrics of synthesis images and anomaly detection performance metrics , and (iv) strategies for hybrid anomaly synthesis methods. Through extensive experiments, ASBench not only reveals limitations in current anomaly synthesis methods but also provides actionable insights for future research directions in anomaly synthesis

Problem

• Evaluating anomaly synthesis methods systematically across diverse datasets and pipelines
• Analyzing synthetic-to-real data ratio impact on anomaly detection performance
• Investigating correlation between synthesis metrics and detection performance metrics

Innovation

• First benchmarking framework for anomaly synthesis evaluation
• Introduces four critical dimensions for systematic assessment
• Reveals limitations and insights for future research