Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification

2025-10-13

AI Summary

Traditional surveys are costly and time-consuming, while purely LLM-synthesized responses often introduce systematic bias. This paper proposes a two-stage “synthesis + correction” framework: first generating large-scale synthetic responses via LLMs, then applying statistical correction—rather than fine-tuning—using a limited set of human-annotated samples. We demonstrate, for the first time, that allocating the majority of human data to correction (not training) significantly improves estimation quality. Our method integrates prompt engineering, lightweight fine-tuning, and panel-data-based bias modeling. Evaluated across nutrition, political, and economic domains, it reduces estimation bias by 24%–86% relative to pure synthesis, increases effective sample size by up to 14%, and maintains estimation bias consistently below 5%. This paradigm offers a novel pathway to low-cost, low-bias large-scale population opinion inference.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Surveys provide valuable insights into public opinion and behavior, but their execution is costly and slow. Large language models (LLMs) have been proposed as a scalable, low-cost substitute for human respondents, but their outputs are often biased and yield invalid estimates. We study the interplay between synthesis methods that use LLMs to generate survey responses and rectification methods that debias population estimates, and explore how human responses are best allocated between them. Using two panel surveys with questions on nutrition, politics, and economics, we find that synthesis alone introduces substantial bias (24-86%), whereas combining it with rectification reduces bias below 5% and increases effective sample size by up to 14%. Overall, we challenge the common practice of using all human responses for fine-tuning, showing that under a fixed budget, allocating most to rectification results in far more effective estimation.

Problem

• Reducing bias in LLM-generated survey responses using rectification methods
• Optimizing human data allocation between synthesis and debiasing techniques
• Improving survey validity by combining prompting, fine-tuning, and rectification

Innovation

• Combining synthesis with rectification reduces bias
• Allocating human responses to rectification improves estimation
• Using prompting and fine-tuning with limited data