ALTBI: Constructing Improved Outlier Detection Models via Optimization of Inlier-Memorization Effect

2024-08-19

arXiv.org

AI Summary

In unsupervised anomaly detection, the scarcity of anomalous samples often causes generative models to overfit to outliers. To address this, we propose ALTBI—a novel framework leveraging the “inlier memory” (IM) effect observed during early training stages of generative models. Theoretically grounded, ALTBI dynamically increases batch size and adaptively truncates loss values to actively suppress gradient updates induced by anomalies. The method is inherently robust, computationally efficient, and compatible with differential privacy guarantees. Evaluated on multiple benchmark datasets, ALTBI achieves state-of-the-art detection performance while significantly accelerating inference. Moreover, under privacy constraints, it demonstrates superior detection stability compared to existing approaches.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Outlier detection (OD) is the task of identifying unusual observations (or outliers) from a given or upcoming data by learning unique patterns of normal observations (or inliers). Recently, a study introduced a powerful unsupervised OD (UOD) solver based on a new observation of deep generative models, called inlier-memorization (IM) effect, which suggests that generative models memorize inliers before outliers in early learning stages. In this study, we aim to develop a theoretically principled method to address UOD tasks by maximally utilizing the IM effect. We begin by observing that the IM effect is observed more clearly when the given training data contain fewer outliers. This finding indicates a potential for enhancing the IM effect in UOD regimes if we can effectively exclude outliers from mini-batches when designing the loss function. To this end, we introduce two main techniques: 1) increasing the mini-batch size as the model training proceeds and 2) using an adaptive threshold to calculate the truncated loss function. We theoretically show that these two techniques effectively filter out outliers from the truncated loss function, allowing us to utilize the IM effect to the fullest. Coupled with an additional ensemble strategy, we propose our method and term it Adaptive Loss Truncation with Batch Increment (ALTBI). We provide extensive experimental results to demonstrate that ALTBI achieves state-of-the-art performance in identifying outliers compared to other recent methods, even with significantly lower computation costs. Additionally, we show that our method yields robust performances when combined with privacy-preserving algorithms.

Problem

• Anomaly Detection
• Normal Data Representation
• Generative Models

Innovation

• Outlier Detection
• Adaptive Loss
• Privacy Preservation