PolicyPulse: LLM-Synthesis Tool for Policy Researchers

2025-04-25

CHI Extended Abstracts

AI Summary

This study addresses the challenges of integrating public opinion and representing diverse perspectives in policy research. We propose the first LLM-driven interactive information synthesis framework specifically designed for policy memo authoring. Methodologically, it integrates topic modeling, native quotation extraction, and alignment verification against authoritative policy reports—enabling structured distillation of public experiences, identification of emergent issues, and inspiration for survey design. Experimental evaluation demonstrates high thematic consistency between generated outputs and official policy documents; validation by 11 policy researchers confirms significant improvements in analytical efficiency, generation of novel interpretive insights, and optimization of survey instrumentation. Our key contribution lies in pioneering the systematic application of LLMs to support closed-loop, pluralistic public input integration in policy analysis. Limitations include insufficient demographic context incorporation and limited factual grounding capabilities—both requiring further enhancement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Application Domains: Humanities & Computational Social Science

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Public opinion shapes policy, yet capturing it effectively to surface diverse perspectives remains challenging. This paper introduces PolicyPulse, an LLM-powered interactive system that synthesizes public experiences from online community discussions to help policy researchers author memos and briefs, leveraging curated real-world anecdotes. Given a specific topic (e.g., “Climate Change”), PolicyPulse returns an organized list of themes (e.g., “Biodiversity Loss” or “Carbon Pricing”), supporting each theme with relevant quotes from real-life anecdotes. We compared PolicyPulse outputs to authoritative policy reports. Additionally, we asked 11 policy researchers across multiple institutions in the Northeastern U.S to compare using PolicyPulse with their expert approach. We found that PolicyPulse’s themes aligned with authoritative reports and helped spark research by analyzing existing data, gathering diverse experiences, revealing unexpected themes, and informing survey or interview design. Participants also highlighted limitations including insufficient demographic context and data verification challenges. Our work demonstrates how AI-powered tools can help influence policy-relevant research and shape policy outcomes.

Problem

• Capturing diverse public opinions for policy research effectively
• Synthesizing public experiences from online discussions for policy briefs
• Aligning AI-generated themes with authoritative policy reports

Innovation

• LLM-powered interactive system for policy synthesis
• Organizes themes with real-life anecdotal quotes
• Compares outputs with authoritative policy reports