Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions

2025-02-24

AI Summary

Existing methods struggle to accurately predict public opinion distributions during early survey design stages. Method: We propose an end-to-end fine-tuning framework for large language models (LLMs) leveraging large-scale structured survey data. Our approach introduces supervised fine-tuning grounded in the joint distribution of demographic subpopulations and their responses—observed empirically in real surveys—and incorporates subpopulation-conditioned modeling alongside cross-survey generalization training. We construct SubPOP, the first large-scale subpopulation–response paired dataset (3.3K survey questions, 70K response samples). Results: Our method reduces the KL divergence between LLM-predicted and human-observed response distributions by up to 46% across diverse subpopulations. It demonstrates strong generalization to unseen surveys and previously unencountered subpopulations, significantly enhancing both the efficiency and inclusivity of early-stage questionnaire design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLMs) present novel opportunities in public opinion research by predicting survey responses in advance during the early stages of survey design. Prior methods steer LLMs via descriptions of subpopulations as LLMs' input prompt, yet such prompt engineering approaches have struggled to faithfully predict the distribution of survey responses from human subjects. In this work, we propose directly fine-tuning LLMs to predict response distributions by leveraging unique structural characteristics of survey data. To enable fine-tuning, we curate SubPOP, a significantly scaled dataset of 3,362 questions and 70K subpopulation-response pairs from well-established public opinion surveys. We show that fine-tuning on SubPOP greatly improves the match between LLM predictions and human responses across various subpopulations, reducing the LLM-human gap by up to 46% compared to baselines, and achieves strong generalization to unseen surveys and subpopulations. Our findings highlight the potential of survey-based fine-tuning to improve opinion prediction for diverse, real-world subpopulations and therefore enable more efficient survey designs. Our code is available at https://github.com/JosephJeesungSuh/subpop.

Problem

• Predict public opinion distributions
• Fine-tune language models
• Improve survey response prediction

Innovation

• Fine-tuning LLMs on survey data
• Uses SubPOP dataset for training
• Reduces LLM-human response gap