Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys

2026-04-19

AI Summary

This study addresses the challenge that synthetic survey responses generated by large language models exhibit highly variable and unpredictable accuracy across questions. To improve estimation precision under a fixed human labeling budget, the authors propose a prediction-augmented inference framework that optimally allocates human respondents. The core contributions include formalizing a quantifiable notion of “correction difficulty,” deriving a closed-form optimal allocation rule for M-estimators—such as multinomial logistic regression and conjoint analysis—and introducing a novel meta-learning-based approach to predict question difficulty without requiring pilot data from the target survey. Experiments on two cross-domain datasets demonstrate that the method captures 61%–79% of the theoretical efficiency gain, reducing mean squared error by 11.4% and 10.5%, respectively.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Metareasoning and Metaheuristics
• Multiagent Systems: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions. We study the design problem of allocating a fixed budget of human respondents across estimation tasks when cheap LLM predictions are available for every task. Our framework combines three components. First, building on Prediction-Powered Inference, we characterize a question-specific rectification difficulty that governs how quickly the estimator's variance decreases with human sample size. Second, we derive a closed-form optimal allocation rule that directs more human labels to tasks where the LLM is least reliable. Third, since rectification difficulty depends on unobserved human responses for new surveys, we propose a meta-learning approach, trained on historical data, that predicts it for entirely new tasks without pilot data. The framework extends to general M-estimation, covering regression coefficients and multinomial logit partworths for conjoint analysis. We validate the framework on two datasets spanning different domains, question types, and LLMs, showing that our approach captures 61-79% of the theoretically attainable efficiency gains, achieving 11.4% and 10.5% MSE reductions without requiring any pilot human data for the target survey.

Problem

• LLM-Augmented Surveys
• Sample Allocation
• Rectification Difficulty
• Prediction-Powered Inference
• Survey Estimation

Innovation

• rectification difficulty
• optimal sample allocation
• prediction-powered inference
• meta-learning
• LLM-augmented surveys