Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection

2024-05-28

arXiv.org

AI Summary

The core challenge in open-world out-of-distribution (OOD) detection lies in the absence of OOD labels and the inherent difficulty of modeling their feature distributions. This paper pioneers the use of neural collapse (NC)—a phenomenon wherein in-distribution (ID) class features collapse onto a low-dimensional subspace in the latent space—for OOD detection. We propose a feature-level separation loss based on orthogonal constraints with respect to the principal ID feature subspace. Unlike confidence-based or data-augmentation-dependent methods, our approach enforces OOD features to be orthogonally projected away from the ID subspace, achieving intrinsic latent-space decoupling between ID and OOD samples. Theoretical analysis and extensive experiments demonstrate significant improvements in FPR95 and AUROC on CIFAR-10, CIFAR-100, and ImageNet, establishing new state-of-the-art performance without requiring output-layer confidence scores or auxiliary data sampling.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

In the open world, detecting out-of-distribution (OOD) data, whose labels are disjoint with those of in-distribution (ID) samples, is important for reliable deep neural networks (DNNs). To achieve better detection performance, one type of approach proposes to fine-tune the model with auxiliary OOD datasets to amplify the difference between ID and OOD data through a separation loss defined on model outputs. However, none of these studies consider enlarging the feature disparity, which should be more effective compared to outputs. The main difficulty lies in the diversity of OOD samples, which makes it hard to describe their feature distribution, let alone design losses to separate them from ID features. In this paper, we neatly fence off the problem based on an aggregation property of ID features named Neural Collapse (NC). NC means that the penultimate features of ID samples within a class are nearly identical to the last layer weight of the corresponding class. Based on this property, we propose a simple but effective loss called Separation Loss, which binds the features of OOD data in a subspace orthogonal to the principal subspace of ID features formed by NC. In this way, the features of ID and OOD samples are separated by different dimensions. By optimizing the feature separation loss rather than purely enlarging output differences, our detection achieves SOTA performance on CIFAR10, CIFAR100 and ImageNet benchmarks without any additional data augmentation or sampling, demonstrating the importance of feature separation in OOD detection. Code is available at https://github.com/Wuyingwen/Pursuing-Feature-Separation-for-OOD-Detection.

Problem

• Enhance out-of-distribution detection
• Separate ID and OOD features
• Utilize Neural Collapse property

Innovation

• Focuses on feature separation for OOD detection
• Utilizes Neural Collapse for feature aggregation
• Introduces orthogonal subspace binding for feature separation