More Parameters Than Populations: A Systematic Literature Review of Large Language Models within Survey Research

2025-09-03

AI Summary

This study addresses core challenges—technical adaptability, data quality, and ethical risks—associated with deploying large language models (LLMs) in survey research. Employing a systematic literature review, it integrates multi-source database keyword searches and citation network analysis to synthesize LLM applications across the survey lifecycle (design → data collection → analysis)—the first such framework-based synthesis. The study proposes a “bidirectional empowerment” paradigm: (1) expanding LLM capabilities in survey-specific tasks—including questionnaire generation, real-time interviewer assistance, and unstructured text analysis; and (2) reciprocally advancing LLM development through domain-knowledge infusion, interpretability enhancement, and bias calibration. It systematically identifies representative use cases, persistent bottlenecks (e.g., validity threats, contextual misalignment), and priority improvement directions at each stage. By bridging survey methodology and AI, the work provides both theoretical foundations and actionable guidance for responsible, effective LLM integration in empirical social science research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Survey research has a long-standing history of being a human-powered field, but one that embraces various technologies for the collection, processing, and analysis of various behavioral, political, and social outcomes of interest, among others. At the same time, Large Language Models (LLMs) bring new technological challenges and prerequisites in order to fully harness their potential. In this paper, we report work-in-progress on a systematic literature review based on keyword searches from multiple large-scale databases as well as citation networks that assesses how LLMs are currently being applied within the survey research process. We synthesize and organize our findings according to the survey research process to include examples of LLM usage across three broad phases: pre-data collection, data collection, and post-data collection. We discuss selected examples of potential use cases for LLMs as well as its pitfalls based on examples from existing literature. Considering survey research has rich experience and history regarding data quality, we discuss some opportunities and describe future outlooks for survey research to contribute to the continued development and refinement of LLMs.

Problem

• Assessing LLM applications in survey research process
• Identifying LLM use cases and pitfalls in surveys
• Exploring survey research's role in LLM development

Innovation

• Systematic literature review methodology
• LLM applications in survey phases
• Data quality integration with LLMs