A Planning Framework for Adaptive Labeling

2025-02-10

AI Summary

This work addresses the low efficiency of acquiring labeled data in high-cost scenarios. We propose a batch-wise adaptive labeling framework formulated as a Markov Decision Process (MDP), which reallocates annotation resources to minimize model prediction uncertainty. To our knowledge, this is the first work to cast adaptive labeling as a tractable, planning-capable MDP. We further introduce Smoothed-Autodiff, a low-variance policy gradient method that achieves superior bias–variance trade-offs, significantly improving training stability. The framework is modular and compatible with off-the-shelf deep learning models and uncertainty quantification modules. Experiments demonstrate that our one-step lookahead policy consistently outperforms mainstream heuristic approaches on both real-world and synthetic datasets. Moreover, Smoothed-Autodiff reduces gradient variance by 3.2× compared to baseline methods, leading to faster convergence and higher final model performance.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Ground truth labels/outcomes are critical for advancing scientific and engineering applications, e.g., evaluating the treatment effect of an intervention or performance of a predictive model. Since randomly sampling inputs for labeling can be prohibitively expensive, we introduce an adaptive labeling framework where measurement effort can be reallocated in batches. We formulate this problem as a Markov decision process where posterior beliefs evolve over time as batches of labels are collected (state transition), and batches (actions) are chosen to minimize uncertainty at the end of data collection. We design a computational framework that is agnostic to different uncertainty quantification approaches including those based on deep learning, and allows a diverse array of policy gradient approaches by relying on continuous policy parameterizations. On real and synthetic datasets, we demonstrate even a one-step lookahead policy can substantially outperform common adaptive labeling heuristics, highlighting the virtue of planning. On the methodological side, we note that standard REINFORCE-style policy gradient estimators can suffer high variance since they rely only on zeroth order information. We propose a direct backpropagation-based approach, Smoothed-Autodiff, based on a carefully smoothed version of the original non-differentiable MDP. Our method enjoys low variance at the price of introducing bias, and we theoretically and empirically show that this trade-off can be favorable.

Problem

• Adaptive labeling framework reduces measurement costs.
• Markov decision process minimizes uncertainty efficiently.
• Smoothed-Autodiff method lowers estimator variance effectively.

Innovation

• Adaptive labeling framework
• Markov decision process
• Smoothed-Autodiff backpropagation