anomaly detection

Detecting unusual patterns using statistical tests, rule engines, unsupervised methods (isolation forest, clustering, one‑class SVM), representation learning (autoencoders) and supervised classifiers for known fraud types, plus thresholding, precision/recall evaluation, concept‑drift monitoring and alerting in production.

anomalydetection

12-Month Skill Trend

Momentum and market value over time
Trending
Score
+20 in 12 mo
96
12 mo agoNow
Career
Value
+$12K in 12 mo
$42K/year
12 mo agoNow

Recommended Survey Paper

Quick overview of the field
View more

Must-Read Papers

Most classic and influential ideas
View more

Explainable Unsupervised Anomaly Detection with Random Forest

Apr 22, 2025
JS
Joshua S. Harvey
🏛️ Prospect 33 | BlackRock

To address low detection accuracy, poor interpretability, and sensitivity to missing values and data preprocessing in unsupervised anomaly detection, this paper proposes a novel Unsupervised Random Forest (URF) framework. URF learns an anisotropic distance metric by discriminatively distinguishing real samples from synthetically generated uniform samples, thereby significantly enhancing anomaly identification near decision boundaries. It is the first work to leverage random forests for unsupervised similarity modeling and localized interpretability: it natively handles missing values and eliminates the need for feature standardization or other preprocessing steps; moreover, it provides feature-level attribution explanations via tree path analysis. Extensive experiments on multiple benchmark datasets demonstrate that URF consistently outperforms state-of-the-art unsupervised detectors, achieving simultaneous improvements in detection accuracy, robustness against data perturbations, and visualization-enabled interpretability.

Developing explainable anomaly predictions via feature importanceHandling missing data without extensive preprocessingImproving unsupervised anomaly detection accuracy

Semi-Supervised Supply Chain Fraud Detection with Unsupervised Pre-Filtering

Aug 07, 2025
FM
Fatemeh Moradi
🏛️ Islamic Azad University | University of Verona | University of Extremadura

To address the challenges of label scarcity and severe class imbalance in supply chain fraud detection, this paper proposes a two-stage semi-supervised learning framework. In the first stage, Isolation Forest performs unsupervised coarse anomaly screening; in the second stage, a self-training SVM refines detection by incorporating high-confidence pseudo-labels. The method innovatively integrates unsupervised anomaly detection with semi-supervised classification. Evaluated on the real-world DataCo supply chain dataset, it achieves an F1-score of 0.817 at a false positive rate below 3.0%, significantly outperforming conventional supervised and single-stage semi-supervised baselines. This work establishes a novel, interpretable, high-accuracy, and deployment-friendly paradigm for supply chain risk control under low-supervision and highly imbalanced conditions.

Address class imbalance and limited labeled dataCombine unsupervised and semi-supervised learning effectivelyDetect fraud in complex global supply chains

Unsupervised time-series anomaly detection often suffers from suboptimal performance when applied directly to raw time-series data due to inadequate feature representation. Method: To address this, we propose an unsupervised feature engineering framework based on tsfresh, which transforms time-series inputs into tabular feature representations; these features are then fed into Isolation Forest (IF) and Local Outlier Factor (LOF) for anomaly detection. Contribution/Results: Extensive experiments on five standard benchmark datasets demonstrate that tsfresh-derived features substantially improve IF’s F1-score (average gain of +12.7%), whereas LOF shows limited improvement. This work provides the first empirical evidence that generic automated feature engineering enhances time-series anomaly detection performance—yet its effectiveness is algorithm-dependent. Crucially, it establishes that learned tabular representations outperform direct modeling of raw time series, underscoring the pivotal role of feature engineering in this task.

Anomaly DetectionData PreprocessingTime Series Analysis

This study addresses the limited generalization capability of existing anomaly detection methods when applied to highly complex, multi-stage industrial time-series data, where capturing aperiodic and multi-scale dynamic patterns remains challenging. For the first time, the authors systematically evaluate a range of unsupervised models—including Isolation Forest, recurrent autoencoders, variational autoencoders, and temporal convolutional autoencoders (TCAE)—in real-world, high-complexity industrial settings. Their experiments reveal significant limitations of classical approaches while demonstrating the superior effectiveness of deep autoencoder architectures. Among these, TCAE exhibits the most robust performance, substantially outperforming competing methods and offering a reliable solution for anomaly detection in intricate industrial processes.

anomaly detectionindustrial time seriesprocess complexity

PIF: Anomaly detection via preference embedding

Jan 10, 2021
FL
Filippo Leveni
🏛️ Politecnico di Milano | Università della Svizzera italiana

This paper addresses anomaly detection in structured data by proposing Preference-based Isolation Forest (PIF), a novel method that maps raw data into a preference-driven high-dimensional embedding space and constructs a PI-Forest tree structure for efficient anomaly scoring. Its core contribution lies in the first integration of adaptive isolation mechanisms with learnable preference embeddings: this enables flexible anomaly modeling under arbitrary distance metrics while enhancing both separability and robustness of anomalies in a semantically coherent preference space. Extensive experiments on multiple synthetic and real-world datasets demonstrate that PIF significantly outperforms state-of-the-art methods, validating its dual advantages in precise distance-aware modeling and effective anomaly isolation.

Combining adaptive isolation with preference embeddingDetecting anomalies in structured patternsMeasuring arbitrary distances in preference space

Latest Papers

What's happening recently
View more

Detecting anomalies in images and video is an essential task for multiple real-world problems, including industrial inspection, computer-assisted diagnosis, and environmental monitoring. Anomaly detection is typically formulated as a one-class classification problem, where the training data consists solely of nominal values, leaving methods built on this assumption susceptible to training label noise. We present a dataset folding method that transforms an arbitrary one-class classifier-based anomaly detector into a fully unsupervised method. This is achieved by making a set of key weak assumptions: that anomalies are uncommon in the training dataset and generally heterogeneous. These assumptions enable us to utilize multiple independently trained instances of a one-class classifier to filter the training dataset for anomalies. This transformation requires no modifications to the underlying anomaly detector; the only changes are algorithmically selected data subsets used for training. We demonstrate that our method can transform a wide variety of one-class classifier anomaly detectors for both images and videos into unsupervised ones. Our method creates the first unsupervised logical anomaly detectors by transforming existing methods. We also demonstrate that our method achieves state-of-the-art performance for unsupervised anomaly detection on the MVTec AD, ViSA, and MVTec Loco AD datasets. As improvements to one-class classifiers are made, our method directly transfers those improvements to the unsupervised domain, linking the domains.

anomaly detectionlabel noiseone-class classification

This study addresses the challenge of detecting anomalous events in large-scale, high-voltage power grid operational data by systematically evaluating the performance of neural networks, k-nearest neighbors, support vector machines, and unsupervised learning methods under complex contextual conditions. The findings reveal that grid anomalies exhibit strong context dependency. Among the evaluated approaches, neural networks significantly outperform traditional methods in overall detection accuracy, while unsupervised learning algorithms demonstrate superior robustness and efficiency in scenarios involving concurrent multiple anomalies. This work not only validates the advantages of deep learning for anomaly detection in power systems but also highlights the practical value of unsupervised methods when labeled data are scarce and fault patterns are intricately coupled.

Anomaly DetectionMachine LearningOperational Data

Industrial anomaly detection is often hindered by the extreme scarcity of fault samples, leading to suboptimal model performance and limited generalization. This work addresses this challenge by constructing a problem-agnostic hyperspherical synthetic dataset to systematically evaluate 14 anomaly detection algorithms—including kNN, LOF, XGBOD, SVM, and CatBoost—under rigorously controlled conditions across varying fault rates (0.05%–20%) and training set sizes. The study quantitatively demonstrates for the first time that unsupervised methods achieve optimal performance when fewer than 20 fault samples are available; semi-supervised and supervised approaches significantly outperform others with 30–50 fault samples; and further increasing normal samples yields diminishing returns. Additionally, feature dimensionality is found to critically influence the efficacy of semi-supervised methods, thereby clarifying the operational boundaries of each algorithmic category.

anomaly detectionclass imbalancefaulty data scarcity

This study addresses the limited effectiveness of unsupervised anomaly detection in identifying fraudulent transactions under label-scarce conditions by proposing an enhanced Isolation Forest method. The approach innovatively integrates the silhouette coefficient into the Isolation Forest framework, leveraging tree path lengths as distinctive “fingerprints” for clustering and fusing silhouette scores with original anomaly scores to improve discriminative power. Designed with adjustable structure and ease of deployment, the method also explicitly delineates its applicability boundaries. Evaluated on the real-world IEEE-CIS dataset, it achieves an average AUC-PR improvement of 0.0080 and consistently outperforms baseline methods across five random trials (p = 0.046). Its limitations are further validated on the Sparkov synthetic dataset.

AUC-PRIsolation Forestsilhouette score

This work addresses the challenge that existing unsupervised methods struggle to reliably detect subtle and noisy anomalies in complex time series, often being misled by noise in normal samples and missing near-normal anomalies. To overcome this limitation, we propose a novel unsupervised anomaly detection framework that integrates active learning: it enhances temporal dependency modeling through a masked time series reconstruction feedback mechanism and employs a minimax optimization strategy to differentially treat normal and anomalous samples, thereby improving robustness against noise and weak anomalies. Extensive experiments across four multivariate time series datasets and seven backbone models demonstrate that our method achieves an average AUC improvement of 12.39%, significantly outperforming current unsupervised approaches.

active learningnoise contaminationsubtle anomalies

Hot Scholars

YC

Yunkang Cao

Hunan University
Visual Anomaly DetectionIndustrial Foundation ModelEmbodied Intelligence
GP

Guansong Pang

Assistant Professor of Computer Science, Singapore Management University
Machine LearningData MiningComputer VisionAnomaly Detection
SP

Shirui Pan

Professor, ARC Future Fellow, FQA, Director of TrustAGI Lab, Griffith University
Data MiningMachine LearningGraph Neural NetworksTrustworthy AI
WS

Weiming Shen

Huazhong University of Science and Technology
JW

Jinbao Wang

Assistant Professor, School of Artificial Intelligence, Shenzhen University
Anomaly DetectionComputer VisionMachine Learning