What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics

2026-06-29

AI Summary

This study addresses the challenge of unsupervised anomaly detection, where the absence of prior knowledge about anomalies makes it difficult to effectively distinguish normal inliers from anomalous outliers. The work provides the first theoretical foundation for the phenomenon that "inlier memory precedes outlier memory" (IM), elucidating its intrinsic connections to data distribution and parameter initialization, and rigorously quantifying its strength and persistence throughout training. Building on these insights, the authors propose an enhanced approach integrating a simplified autoencoder architecture, dynamic analysis of training trajectories, tailored data preprocessing, and strategic initialization, yielding a practical optimization guideline. Extensive experiments on the ADBench benchmark demonstrate state-of-the-art performance, confirming both the effectiveness and practical utility of the proposed method.

Abstract

Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods.

Problem

• outlier detection
• inlier-memorization effect
• unsupervised learning
• early training dynamics
• theoretical understanding

Innovation

• inlier-memorization effect
• outlier detection
• early training dynamics
• autoencoder
• unsupervised anomaly detection