Revisiting Active Sequential Prediction-Powered Mean Estimation

2026-04-20

AI Summary

This study addresses the problem of active sequential mean estimation, where the goal is to dynamically decide whether to query true labels in order to balance estimation accuracy against querying cost. The work proposes a method that integrates covariate-aware prediction with selective label querying, adaptively choosing between model-based prediction and acquiring ground-truth labels based on a prescribed query probability after observing covariates. The theoretical contributions include establishing non-asymptotic, data-dependent confidence intervals; proving that under no-regret learning, the optimal querying policy converges to the maximum allowable query probability independent of covariates; and revealing that a constant high query probability—close to one—typically yields the narrowest confidence intervals. Simulations corroborate these findings and demonstrate the effectiveness of the proposed uncertainty-guided querying strategy.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Sequential Decision Making
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

In this work, we revisit the problem of active sequential prediction-powered mean estimation, where at each round one must decide the query probability of the ground-truth label upon observing the covariates of a sample. Furthermore, if the label is not queried, the prediction from a machine learning model is used instead. Prior work proposed an elegant scheme that determines the query probability by combining an uncertainty-based suggestion with a constant probability that encodes a soft constraint on the query probability. We explored different values of the mixing parameter and observed an intriguing empirical pattern: the smallest confidence width tends to occur when the weight on the constant probability is close to one, thereby reducing the influence of the uncertainty-based component. Motivated by this observation, we develop a non-asymptotic analysis of the estimator and establish a data-dependent bound on its confidence interval. Our analysis further suggests that when a no-regret learning approach is used to determine the query probability and control this bound, the query probability converges to the constraint of the max value of the query probability when it is chosen obliviously to the current covariates. We also conduct simulations that corroborate these theoretical findings.

Problem

• active sequential prediction
• mean estimation
• query probability
• uncertainty-based sampling
• confidence interval

Innovation

• active sequential estimation
• prediction-powered inference
• non-asymptotic analysis
• no-regret learning
• query probability