Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data

2026-06-27

AI Summary

This study addresses systematic biases in large language models (LLMs) when simulating survey responses, including skewed marginal distributions, poor variance calibration, and attenuated variable relationships. It proposes a novel decomposition of simulation fidelity into three quantifiable dimensions—structural, marginal, and individual—and systematically evaluates the multidimensional fidelity of three mitigation strategies: prompt engineering, output post-processing, and few-shot fine-tuning, using small-scale pilot data. Empirical results demonstrate that few-shot fine-tuning achieves a favorable balance across fidelity dimensions; however, uneven fidelity across subpopulations may compromise the consistent alignment of diverse viewpoints.

Abstract

Large language models (LLMs) are increasingly used to simulate social survey responses, yet their outputs exhibit systematic biases: marginal distributions are skewed, response variance is poorly calibrated, and predictor-outcome relationships are attenuated. We ask a simple question: given a small pilot sample of human responses, can an LLM recover the statistical characteristics of a broader population? We decompose recovery along three axes: structural fidelity, marginal fidelity, and individual fidelity. Using a COVID-19 misinformation survey as a case study, we benchmark three families of approaches: prompting, rectification, and fine-tuning. The findings suggest that fine-tuning on small pilot samples offers a balanced approach for achieving multiple forms of fidelity, but the levels of such fidelity can vary across subsamples, potentially threatening pluralistic alignment.

Problem

• LLM-based survey simulation
• statistical fidelity
• systematic bias
• small pilot data
• pluralistic alignment

Innovation

• three-axis fidelity
• LLM-based survey simulation
• small pilot data
• statistical alignment
• fine-tuning