Leveraging Intermediate Representations for Better Out-of-Distribution Detection

2025-02-18

AI Summary

In real-world applications, reliable out-of-distribution (OoD) detection is critical for ensuring the safety and robustness of machine learning decisions. Existing methods predominantly rely on final-layer outputs, overlooking discriminative information encoded in intermediate representations. This work systematically demonstrates, for the first time, that intermediate-layer activations exhibit strong OoD discrimination capability. We propose an energy-driven contrastive regularization strategy, a multi-layer feature aggregation mechanism, and an OoD scoring function based on activation statistics. Our approach breaks the conventional single-layer dependency paradigm and fully exploits the hierarchical representational capacity of deep networks. Evaluated on multiple standard benchmarks, our method achieves an average 12.3% reduction in false positive rate at 95% true positive rate (FPR95), while preserving in-distribution classification accuracy—outperforming state-of-the-art OoD detection methods. These results validate both the effectiveness and generalizability of leveraging intermediate representations for OoD detection.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Representation Learning for Vision
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In real-world applications, machine learning models must reliably detect Out-of-Distribution (OoD) samples to prevent unsafe decisions. Current OoD detection methods often rely on analyzing the logits or the embeddings of the penultimate layer of a neural network. However, little work has been conducted on the exploitation of the rich information encoded in intermediate layers. To address this, we analyze the discriminative power of intermediate layers and show that they can positively be used for OoD detection. Therefore, we propose to regularize intermediate layers with an energy-based contrastive loss, and by grouping multiple layers in a single aggregated response. We demonstrate that intermediate layer activations improves OoD detection performance by running a comprehensive evaluation across multiple datasets.

Problem

• Detect Out-of-Distribution samples reliably
• Exploit intermediate layers' information
• Improve OoD detection with regularization

Innovation

• Uses intermediate layer activations
• Applies energy-based contrastive loss
• Aggregates multiple layers response