Refining Decision Boundaries In Anomaly Detection Using Similarity Search Within the Feature Space

2026-02-02

AI Summary

This work addresses the challenge of detecting rare and diverse anomalies in highly imbalanced data, such as APT attacks, by proposing SDA2E—a novel autoencoder that integrates sparse binary embeddings with an adversarial attention mechanism. To further enhance detection performance, the authors design an active learning framework based on a similarity metric called Normalized Matching 1s (NM1). This framework efficiently refines the anomaly decision boundary in feature space through normal-like expansion, anomaly-like prioritization, and hybrid querying strategies. Extensive experiments across 52 imbalanced datasets demonstrate that SDA2E significantly outperforms 15 state-of-the-art methods, achieving an nDCG as high as 1.0, reducing labeling requirements by up to 80%, and exhibiting statistically significant performance gains.

Technology Category

• Machine Learning: Active Learning
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Detecting rare and diverse anomalies in highly imbalanced datasets-such as Advanced Persistent Threats (APTs) in cybersecurity-remains a fundamental challenge for machine learning systems. Active learning offers a promising direction by strategically querying an oracle to minimize labeling effort, yet conventional approaches often fail to exploit the intrinsic geometric structure of the feature space for model refinement. In this paper, we introduce SDA2E, a Sparse Dual Adversarial Attention-based AutoEncoder designed to learn compact and discriminative latent representations from imbalanced, high-dimensional data. We further propose a similarity-guided active learning framework that integrates three novel strategies to refine decision boundaries efficiently: mormal-like expansion, which enriches the training set with points similar to labeled normals to improve reconstruction fidelity; anomaly-like prioritization, which boosts ranking accuracy by focusing on points resembling known anomalies; and a hybrid strategy that combines both for balanced model refinement and ranking. A key component of our framework is a new similarity measure, Normalized Matching 1s (SIM_NM1), tailored for sparse binary embeddings. We evaluate SDA2E extensively across 52 imbalanced datasets, including multiple DARPA Transparent Computing scenarios, and benchmark it against 15 state-of-the-art anomaly detection methods. Results demonstrate that SDA2E consistently achieves superior ranking performance (nDCG up to 1.0 in several cases) while reducing the required labeled data by up to 80% compared to passive training. Statistical tests confirm the significance of these improvements. Our work establishes a robust, efficient, and statistically validated framework for anomaly detection that is particularly suited to cybersecurity applications such as APT detection.

Problem

• anomaly detection
• imbalanced datasets
• Advanced Persistent Threats
• decision boundaries
• cybersecurity

Innovation

• active learning
• anomaly detection
• similarity search
• sparse autoencoder
• imbalanced data