ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection

2024-02-27

International Conference on Learning Representations

AI Summary

Existing post-hoc out-of-distribution (OOD) detection methods rely on logits, distance metrics, or strong distributional assumptions, limiting their ability to accurately model true data density. To address this, we propose a unified density modeling framework based on Bregman divergences, reformulating density estimation as a differentiable optimization problem for the optimal norm coefficient $ p $. We first uncover a novel paradigm for exponential-family density modeling under conjugate constraints, leading to ConjNorm—a method that achieves unbiased, analytically differentiable density estimation without restrictive distributional assumptions. ConjNorm integrates Bregman divergence theory, exponential-family modeling, and Monte Carlo importance sampling. On CIFAR-100 and ImageNet-1K, it reduces false positive rate at 95% true positive rate (FPR95) by 13.25% and 28.19%, respectively, over prior state-of-the-art methods, significantly improving both OOD detection accuracy and robustness.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Post-hoc out-of-distribution (OOD) detection has garnered intensive attention in reliable machine learning. Many efforts have been dedicated to deriving score functions based on logits, distances, or rigorous data distribution assumptions to identify low-scoring OOD samples. Nevertheless, these estimate scores may fail to accurately reflect the true data density or impose impractical constraints. To provide a unified perspective on density-based score design, we propose a novel theoretical framework grounded in Bregman divergence, which extends distribution considerations to encompass an exponential family of distributions. Leveraging the conjugation constraint revealed in our theorem, we introduce a extsc{ConjNorm} method, reframing density function design as a search for the optimal norm coefficient $p$ against the given dataset. In light of the computational challenges of normalization, we devise an unbiased and analytically tractable estimator of the partition function using the Monte Carlo-based importance sampling technique. Extensive experiments across OOD detection benchmarks empirically demonstrate that our proposed extsc{ConjNorm} has established a new state-of-the-art in a variety of OOD detection setups, outperforming the current best method by up to 13.25$%$ and 28.19$%$ (FPR95) on CIFAR-100 and ImageNet-1K, respectively.

Problem

• Developing tractable density estimation for out-of-distribution detection
• Overcoming computational challenges in partition function estimation
• Providing unified theoretical framework for density-based score design

Innovation

• Framework uses Bregman divergence for exponential family distributions
• ConjNorm method finds optimal norm coefficient p for datasets
• Monte Carlo importance sampling estimates partition function tractably