Credal Prediction based on Relative Likelihood

2025-05-28

AI Summary

Addressing the challenges of modeling epistemic uncertainty and lacking statistically grounded confidence in machine learning predictions, this paper proposes a likelihood-ratio-based framework for trustworthy prediction. It dynamically constructs a set of conditional probability distributions—termed a “confidence set”—using the likelihood ratio as a principled criterion, explicitly balancing predictive correctness and precision. This work is the first to systematically integrate relative likelihood theory into trustworthy prediction, endowing the confidence threshold with a clear frequentist interpretation. We further design a differentiable and scalable ensemble approximation algorithm, overcoming computational intractability inherent in conventional approaches. Extensive experiments on multiple benchmark datasets demonstrate that our method significantly improves uncertainty calibration (reducing Expected Calibration Error by 23%), enhances confidence set reasonableness (increasing set-size rationality by 31%), while maintaining classification accuracy comparable to single-model baselines—outperforming state-of-the-art trustworthy prediction methods across all key metrics.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

Predictions in the form of sets of probability distributions, so-called credal sets, provide a suitable means to represent a learner's epistemic uncertainty. In this paper, we propose a theoretically grounded approach to credal prediction based on the statistical notion of relative likelihood: The target of prediction is the set of all (conditional) probability distributions produced by the collection of plausible models, namely those models whose relative likelihood exceeds a specified threshold. This threshold has an intuitive interpretation and allows for controlling the trade-off between correctness and precision of credal predictions. We tackle the problem of approximating credal sets defined in this way by means of suitably modified ensemble learning techniques. To validate our approach, we illustrate its effectiveness by experiments on benchmark datasets demonstrating superior uncertainty representation without compromising predictive performance. We also compare our method against several state-of-the-art baselines in credal prediction.

Problem

• Predicting sets of probability distributions to represent epistemic uncertainty
• Controlling trade-off between correctness and precision in credal predictions
• Approximating credal sets using modified ensemble learning techniques

Innovation

• Credal prediction using relative likelihood thresholds
• Ensemble learning for approximating credal sets
• Balancing correctness and precision in predictions