Labels or Preferences? Budget-Constrained Learning with Human Judgments over AI-Generated Outputs

2026-01-19

AI Summary

This study addresses the critical challenge of optimally allocating limited human annotation budgets between ground-truth labels and pairwise preferences over AI outputs to enhance model evaluation efficiency. The work formalizes this budget allocation problem as a statistical estimation task and introduces Preference-Calibrated Active Learning (PCAL), a novel method grounded in semiparametric inference and monotone missing data modeling. PCAL directly minimizes the variance of functional estimators without requiring closed-form solutions and exhibits robustness to perturbations in the underlying model. Theoretically, the PCAL estimator is shown to be asymptotically optimal. Extensive experiments on both synthetic and real-world datasets demonstrate that PCAL significantly outperforms existing approaches under constrained annotation budgets.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Probabilistic Inference

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

The increasing reliance on human preference feedback to judge AI-generated pseudo labels has created a pressing need for principled, budget-conscious data acquisition strategies. We address the crucial question of how to optimally allocate a fixed annotation budget between ground-truth labels and pairwise preferences in AI. Our solution, grounded in semi-parametric inference, casts the budget allocation problem as a monotone missing data framework. Building on this formulation, we introduce Preference-Calibrated Active Learning (PCAL), a novel method that learns the optimal data acquisition strategy and develops a statistically efficient estimator for functionals of the data distribution. Theoretically, we prove the asymptotic optimality of our PCAL estimator and establish a key robustness guarantee that ensures robust performance even with poorly estimated nuisance models. Our flexible framework applies to a general class of problems, by directly optimizing the estimator's variance instead of requiring a closed-form solution. This work provides a principled and statistically efficient approach for budget-constrained learning in modern AI. Simulations and real-data analysis demonstrate the practical benefits and superior performance of our proposed method.

Problem

• budget-constrained learning
• human judgments
• preference feedback
• data acquisition
• AI-generated outputs

Innovation

• Preference-Calibrated Active Learning
• budget-constrained learning
• semi-parametric inference
• missing data framework
• statistical efficiency