Evaluating Anomaly Detectors for Simulated Highly Imbalanced Industrial Classification Problems

2025-12-07

arXiv.org

AI Summary

Industrial anomaly detection is often hindered by the extreme scarcity of fault samples, leading to suboptimal model performance and limited generalization. This work addresses this challenge by constructing a problem-agnostic hyperspherical synthetic dataset to systematically evaluate 14 anomaly detection algorithms—including kNN, LOF, XGBOD, SVM, and CatBoost—under rigorously controlled conditions across varying fault rates (0.05%–20%) and training set sizes. The study quantitatively demonstrates for the first time that unsupervised methods achieve optimal performance when fewer than 20 fault samples are available; semi-supervised and supervised approaches significantly outperform others with 30–50 fault samples; and further increasing normal samples yields diminishing returns. Additionally, feature dimensionality is found to critically influence the efficacy of semi-supervised methods, thereby clarifying the operational boundaries of each algorithmic category.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Algorithm Configuration

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Machine learning offers potential solutions to current issues in industrial systems in areas such as quality control and predictive maintenance, but also faces unique barriers in industrial applications. An ongoing challenge is extreme class imbalance, primarily due to the limited availability of faulty data during training. This paper presents a comprehensive evaluation of anomaly detection algorithms using a problem-agnostic simulated dataset that reflects real-world engineering constraints. Using a synthetic dataset with a hyper-spherical based anomaly distribution in 2D and 10D, we benchmark 14 detectors across training datasets with anomaly rates between 0.05% and 20% and training sizes between 1 000 and 10 000 (with a testing dataset size of 40 000) to assess performance and generalization error. Our findings reveal that the best detector is highly dependant on the total number of faulty examples in the training dataset, with additional healthy examples offering insignificant benefits in most cases. With less than 20 faulty examples, unsupervised methods (kNN/LOF) dominate; but around 30-50 faulty examples, semi-supervised (XGBOD) and supervised (SVM/CatBoost) detectors, we see large performance increases. While semi-supervised methods do not show significant benefits with only two features, the improvements are evident at ten features. The study highlights the performance drop on generalization of anomaly detection methods on smaller datasets, and provides practical insights for deploying anomaly detection in industrial environments.

Problem

• anomaly detection
• class imbalance
• industrial classification
• faulty data scarcity
• generalization error

Innovation

• anomaly detection
• class imbalance
• synthetic dataset
• semi-supervised learning
• industrial applications