Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations

2025-02-10

AI Summary

High cost and long duration hinder large-scale social surveys. Method: This paper proposes the first LLM specialization method for simulating national-level population response distributions to survey questions, centered on distribution alignment via first-token probability calibration—optimized by minimizing KL divergence between model outputs and empirical response distributions—without requiring question-level annotations. Contribution/Results: The method generalizes across questions, countries, and surveys. Evaluated on multi-source cultural survey datasets—including the World Values Survey and European Values Study—it achieves superior zero-shot performance over baseline classifiers and other methods on unseen questions, new countries, and entirely novel surveys. Results demonstrate that distribution-level specialization significantly enhances both effectiveness and robustness in modeling sociobehavioral response patterns.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large-scale surveys are essential tools for informing social science research and policy, but running surveys is costly and time-intensive. If we could accurately simulate group-level survey results, this would therefore be very valuable to social science research. Prior work has explored the use of large language models (LLMs) for simulating human behaviors, mostly through prompting. In this paper, we are the first to specialize LLMs for the task of simulating survey response distributions. As a testbed, we use country-level results from two global cultural surveys. We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions for a given question. Then, we show that this method substantially outperforms other methods and zero-shot classifiers, even on unseen questions, countries, and a completely unseen survey. While even our best models struggle with the task, especially on unseen questions, our results demonstrate the benefits of specialization for simulation, which may accelerate progress towards sufficiently accurate simulation in the future.

Problem

• Simulating survey response distributions globally
• Specializing LLMs for accurate survey simulations
• Improving simulation accuracy on unseen questions

Innovation

• Specializing LLMs for surveys
• Fine-tuning with first-token probabilities
• Outperforming zero-shot classifiers