Kernel PCA for Out-of-Distribution Detection

2024-02-05

arXiv.org

AI Summary

Detecting out-of-distribution (OoD) samples in deep neural networks remains challenging—especially where linear methods like PCA lack sufficient discriminative power. Method: This paper proposes an efficient nonlinear subspace modeling approach based on kernel principal component analysis (KPCA) for OoD detection. It introduces, for the first time in OoD detection, an explicit-feature-mapping KPCA framework that balances theoretical interpretability with scalability. A task-adaptive multi-scale kernel function is designed to enhance separability between in-distribution and OoD samples in the implicit feature space, and a lightweight detection criterion is formulated using reconstruction error. Results: The method achieves state-of-the-art performance across multiple benchmark datasets and mainstream architectures, significantly outperforming PCA and existing OoD approaches in detection accuracy while maintaining low computational overhead.

Technology Category

• Machine Learning: Kernel Methods
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Other Foundations of Computer Vision

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Out-of-Distribution (OoD) detection is vital for the reliability of Deep Neural Networks (DNNs). Existing works have shown the insufficiency of Principal Component Analysis (PCA) straightforwardly applied on the features of DNNs in detecting OoD data from In-Distribution (InD) data. The failure of PCA suggests that the network features residing in OoD and InD are not well separated by simply proceeding in a linear subspace, which instead can be resolved through proper non-linear mappings. In this work, we leverage the framework of Kernel PCA (KPCA) for OoD detection, and seek suitable non-linear kernels that advocate the separability between InD and OoD data in the subspace spanned by the principal components. Besides, explicit feature mappings induced from the devoted task-specific kernels are adopted so that the KPCA reconstruction error for new test samples can be efficiently obtained with large-scale data. Extensive theoretical and empirical results on multiple OoD data sets and network structures verify the superiority of our KPCA detector in efficiency and efficacy with state-of-the-art detection performance.

Problem

• Anomaly Detection
• Deep Neural Networks
• Principal Component Analysis

Innovation

• Kernel PCA
• Deep Neural Networks
• Anomaly Detection