Credal Ensemble Distillation for Uncertainty Quantification

2025-11-14

AI Summary

Deep ensembles (DEs) effectively distinguish aleatoric from epistemic uncertainty but incur prohibitive inference overhead, hindering practical deployment. To address this, we propose the Credible Set Distillation (CED) framework, which compresses a DE into a single lightweight model—CREDIT—while preserving uncertainty-aware prediction capabilities. Crucially, CED introduces credible set modeling into knowledge distillation: instead of distilling point-wise class probabilities, it transfers *class-level probability intervals*, explicitly retaining the ability to discriminate uncertainty types. Our approach integrates convex probabilistic set modeling with uncertainty decomposition theory, enabling efficient and principled uncertainty quantification in classification. Experiments demonstrate that CREDIT matches or surpasses state-of-the-art methods in out-of-distribution detection, while reducing inference computation and memory footprint by orders of magnitude—significantly enhancing deployability without sacrificing reliability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Deep ensembles (DE) have emerged as a powerful approach for quantifying predictive uncertainty and distinguishing its aleatoric and epistemic components, thereby enhancing model robustness and reliability. However, their high computational and memory costs during inference pose significant challenges for wide practical deployment. To overcome this issue, we propose credal ensemble distillation (CED), a novel framework that compresses a DE into a single model, CREDIT, for classification tasks. Instead of a single softmax probability distribution, CREDIT predicts class-wise probability intervals that define a credal set, a convex set of probability distributions, for uncertainty quantification. Empirical results on out-of-distribution detection benchmarks demonstrate that CED achieves superior or comparable uncertainty estimation compared to several existing baselines, while substantially reducing inference overhead compared to DE.

Problem

• Compressing deep ensembles into single models for efficient uncertainty quantification
• Reducing computational and memory costs of ensemble inference in classification tasks
• Predicting class-wise probability intervals instead of single distributions for uncertainty estimation

Innovation

• Compresses deep ensembles into single model
• Predicts class-wise probability intervals for uncertainty
• Reduces inference overhead while maintaining performance