Estimating prevalence with precision and accuracy

2025-07-08

AI Summary

In class distribution estimation (quantification), persistent challenges include training-set prevalence bias and reliable uncertainty quantification—particularly in terms of confidence interval width, coverage accuracy, and calibration. To address these, we propose Precise Quantifier (PQ), a novel Bayesian quantification framework that explicitly models key factors such as classifier discriminative capability, labeled and unlabeled dataset sizes, and label noise. PQ jointly optimizes bias correction and uncertainty calibration through principled posterior inference. Theoretical analysis establishes PQ’s consistency and asymptotic normality under mild assumptions. Extensive experiments across diverse real-world and synthetic benchmarks demonstrate that PQ yields significantly narrower confidence intervals while achieving near-perfect coverage calibration—outperforming classical bootstrap-based approaches and state-of-the-art Bayesian quantifiers. This work introduces a new paradigm for distribution inference that simultaneously delivers high estimation precision and well-calibrated, reliable uncertainty quantification.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Unlike classification, whose goal is to estimate the class of each data point in a dataset, prevalence estimation or quantification is a task that aims to estimate the distribution of classes in a dataset. The two main tasks in prevalence estimation are to adjust for bias, due to the prevalence in the training dataset, and to quantify the uncertainty in the estimate. The standard methods used to quantify uncertainty in prevalence estimates are bootstrapping and Bayesian quantification methods. It is not clear which approach is ideal in terms of precision (i.e. the width of confidence intervals) and coverage (i.e. the confidence intervals being well-calibrated). Here, we propose Precise Quantifier (PQ), a Bayesian quantifier that is more precise than existing quantifiers and with well-calibrated coverage. We discuss the theory behind PQ and present experiments based on simulated and real-world datasets. Through these experiments, we establish the factors which influence quantification precision: the discriminatory power of the underlying classifier; the size of the labeled dataset used to train the quantifier; and the size of the unlabeled dataset for which prevalence is estimated. Our analysis provides deep insights into uncertainty quantification for quantification learning.

Problem

• Estimating class distribution in datasets accurately
• Comparing uncertainty methods for prevalence estimation
• Improving precision and coverage in Bayesian quantifiers

Innovation

• Proposes Precise Quantifier (PQ) Bayesian method
• Compares precision and coverage of uncertainty methods
• Analyzes factors affecting quantification precision