GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detection

2025-05-21

AI Summary

This work addresses the challenge of improving robust out-of-distribution (OOD) detection for neural networks in standard image classification. Existing methods exhibit inconsistent performance, prompting a theory-driven spectral approach: we first unify the neural tangent kernel (NTK) alignment property with the low-rank structure of class-wise gradient means, then construct an efficient and interpretable OOD detector via principal component analysis (PCA) on these gradient class means. Theoretical analysis identifies pre-trained feature quality as the key determinant of OOD detection performance. Empirically, our method achieves state-of-the-art results across multiple benchmarks, demonstrating superior generalization and stability compared to prior approaches. It establishes a novel paradigm for spectral-analysis-based OOD detection, offering both theoretical grounding and practical efficacy.

Technology Category

• Machine Learning: Kernel Methods
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce GradPCA, an Out-of-Distribution (OOD) detection method that exploits the low-rank structure of neural network gradients induced by Neural Tangent Kernel (NTK) alignment. GradPCA applies Principal Component Analysis (PCA) to gradient class-means, achieving more consistent performance than existing methods across standard image classification benchmarks. We provide a theoretical perspective on spectral OOD detection in neural networks to support GradPCA, highlighting feature-space properties that enable effective detection and naturally emerge from NTK alignment. Our analysis further reveals that feature quality -- particularly the use of pretrained versus non-pretrained representations -- plays a crucial role in determining which detectors will succeed. Extensive experiments validate the strong performance of GradPCA, and our theoretical framework offers guidance for designing more principled spectral OOD detectors.

Problem

• Detects Out-of-Distribution samples using NTK-aligned gradients
• Improves OOD detection consistency via PCA on gradient class-means
• Analyzes feature-space properties for effective spectral OOD detection

Innovation

• Uses PCA on gradient class-means
• Leverages NTK alignment for OOD detection
• Analyzes feature-space properties theoretically