How can LLMs Support Policy Researchers? Evaluating an LLM-Assisted Workflow for Large-Scale Unstructured Data

2026-04-06

AI Summary

This study addresses the challenge policy researchers face in efficiently extracting public opinion from large-scale unstructured text, as traditional qualitative methods are costly and lack scalability. The authors propose and evaluate a human-in-the-loop workflow leveraging large language models (LLMs) to conduct automated thematic analysis on millions of texts—including Reddit posts and transcripts from chatbot interviews—in a real-world policy research setting. Deployed as a rapid exploratory tool by policy analysts, the approach yields insights that align with, yet also diverge from, findings in authoritative policy reports on key issues. The results systematically demonstrate the practical utility, potential, and limitations of LLMs for public opinion mining in policy contexts.

Abstract

Policy researchers need scalable ways to surface public views, yet they often rely on interviews, listening sessions, and surveys-analyzed thematically-that are slow, expensive, and limited in scale and diversity. LLMs offer new possibilities for thematic analysis of unstructured text, yet we know little about how LLM-assisted workflows perform for policy research. Building on a workflow for LLM-assisted thematic analysis of online forums, we conduct a study with 11 policy researchers, who use an early prototype and see it as a quick, rough-and-ready input to their research. We then extend and scale the workflow to analyze millions of Reddit posts and 1,058 chatbot-led interview transcripts on a policy-relevant topic, treating these sources as rich and scalable data for policy discourse. We compare the synthesized themes to those from authoritative policy reports, identify points of alignment and divergence, and discuss what this implies for policy researchers adopting LLM-assisted workflows.

Problem

• policy research
• large language models
• thematic analysis
• unstructured data
• scalability

Innovation

• LLM-assisted thematic analysis
• scalable policy research
• unstructured text analysis
• Reddit data
• chatbot-led interviews