Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the role of model complexity

2024-11-04

arXiv.org

AI Summary

This work investigates the impact of model complexity—particularly over-parameterization—on post-hoc out-of-distribution (OOD) detection performance. We empirically discover and theoretically characterize the Double Descent phenomenon in OOD detection for the first time. Through bias-variance decomposition and generalization bound analysis, complemented by large-scale empirical evaluation across architectures, datasets, and OOD detectors—as well as systematic complexity-sweep experiments—we establish the ubiquity of this phenomenon and demonstrate that over-parameterization does not universally outperform under-parameterization. Building on these findings, we propose a complexity inflection-point-based method to identify an optimal detection regime, improving AUROC by an average of +2.3%. Our core contributions are threefold: (i) establishing the first theoretical linkage between Double Descent and OOD detection performance; (ii) revealing its non-monotonic dependence on model complexity; and (iii) providing a transferable, complexity-adaptive optimization strategy for OOD detection.

Technology Category

• Machine Learning: Optimization
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Out-of-distribution (OOD) detection is essential for ensuring the reliability and safety of machine learning systems. In recent years, it has received increasing attention, particularly through post-hoc detection and training-based methods. In this paper, we focus on post-hoc OOD detection, which enables identifying OOD samples without altering the model's training procedure or objective. Our primary goal is to investigate the relationship between model capacity and its OOD detection performance. Specifically, we aim to answer the following question: Does the Double Descent phenomenon manifest in post-hoc OOD detection? This question is crucial, as it can reveal whether overparameterization, which is already known to benefit generalization, can also enhance OOD detection. Despite the growing interest in these topics by the classic supervised machine learning community, this intersection remains unexplored for OOD detection. We empirically demonstrate that the Double Descent effect does indeed appear in post-hoc OOD detection. Furthermore, we provide theoretical insights to explain why this phenomenon emerges in such setting. Finally, we show that the overparameterized regime does not yield superior results consistently, and we propose a method to identify the optimal regime for OOD detection based on our observations.

Problem

• Investigates model capacity impact on OOD detection performance
• Explores Double Descent phenomenon in post-hoc OOD detection
• Determines optimal model regime for effective OOD detection

Innovation

• Post-hoc OOD detection without altering training
• Double Descent effect in OOD detection
• Optimal regime identification for OOD performance