Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses

2026-05-18

AI Summary

This study addresses structural challenges in survey research—including declining response rates, sample bias, block-wise missing data among high-risk populations, and AI-generated fraudulent responses—by proposing a five-stage LLM-integrated framework spanning questionnaire design, sampling, pretesting, missing data imputation, and post-survey analysis. The authors innovatively develop an anchor-marginal theory-guided A-TLM model that incorporates a knowledge graph constrained by Protection Motivation Theory, enabling synergistic structured retrieval and single-step inference, alongside a novel grouped bias auditing criterion. Empirical evaluation using 2024 Florida Hurricane Milton preparedness survey data demonstrates that under block-wise missing-not-at-random (MNAR) conditions, A-TLM achieves an RMSE of 1.439—significantly outperforming conventional imputation methods—and exhibits near-zero sign bias (−0.121), markedly better than random forest imputation (−0.631).

Abstract

Survey research faces mounting structural challenges: declining response rates, sample bias, block-wise missingness among at-risk respondents, and AI-assisted fraudulent completions in online panels. Large language models (LLMs) have been proposed as a remedy, yet rigorous evaluations across the full survey workflow remain scarce, particularly in disaster contexts where data quality matters most. We present and evaluate a five-stage framework for LLM integration covering questionnaire design, sample selection, pilot testing, missing-data imputation, and post-collection analysis, using the 2024 Hurricane Milton preparedness survey of Florida residents (n=946) as a shared empirical testbed. We introduce a Protection Motivation Theory (PMT)-constrained co-occurrence knowledge graph and develop seven LLM configurations spanning zero-shot inference, retrieval-augmented baselines, and novel theory-informed variants. Our proposed Anchored Marginal Theory-Informed LLM (A-TLM) outperforms all three classical imputation baselines (IPW/MI, MICE+PMM, missForest) on RMSE under disaster-relevant block-wise MNAR conditions (S4 RMSE 1.439 vs. 1.496 for the next-best), while achieving near-zero signed bias (-0.121) where the random-forest imputer produces the largest absolute bias (-0.631). Organizing retrieval around PMT causal structure and integrating all evidence in a single model call outperforms unstructured retrieval and staged sequential inference (MAE 0.993 vs. 1.097 for standard RAG). We document that near-zero aggregate bias can mask opposing subgroup errors and propose subgroup-stratified bias auditing as a reporting standard. A retrieval-constrained knowledge-graph chatbot demonstrates that hallucination is architecturally manageable through grounded refusal.

Problem

• survey research
• response rates
• sample bias
• missing data
• AI-assisted fraud

Innovation

• Large Language Models
• Survey Research
• Missing Data Imputation
• Protection Motivation Theory
• Knowledge Graph