Applying Large Language Models to Characterize Public Narratives

2025-11-17

AI Summary

This study addresses the subjectivity and high annotation cost associated with expert-driven qualitative analysis of public narratives (PNs). We propose the first LLM-driven analytical framework that systematically integrates a domain-expert-developed codebook. Methodologically, we co-designed a fine-grained coding scheme with domain experts to guide LLM-based qualitative annotation, achieving an average F1-score of 0.80 across eight narrative types and fourteen coding dimensions—comparable to human expert performance. We further demonstrate cross-domain transferability by applying the framework to political speech analysis, successfully uncovering structural patterns in 22 narrative texts. Our contributions are threefold: (1) the first deep integration of LLMs with expert-derived coding schemes for rigorous qualitative analysis; (2) empirical validation of strong generalizability across distinct narrative domains; and (3) the establishment of a novel computational paradigm for citizen narrative research grounded in theoretically informed, scalable annotation.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Application Domains: Humanities & Computational Social Science
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Public Narratives (PNs) are key tools for leadership development and civic mobilization, yet their systematic analysis remains challenging due to their subjective interpretation and the high cost of expert annotation. In this work, we propose a novel computational framework that leverages large language models (LLMs) to automate the qualitative annotation of public narratives. Using a codebook we co-developed with subject-matter experts, we evaluate LLM performance against that of expert annotators. Our work reveals that LLMs can achieve near-human-expert performance, achieving an average F1 score of 0.80 across 8 narratives and 14 codes. We then extend our analysis to empirically explore how PN framework elements manifest across a larger dataset of 22 stories. Lastly, we extrapolate our analysis to a set of political speeches, establishing a novel lens in which to analyze political rhetoric in civic spaces. This study demonstrates the potential of LLM-assisted annotation for scalable narrative analysis and highlights key limitations and directions for future research in computational civic storytelling.

Problem

• Automating qualitative annotation of public narratives using large language models
• Systematically analyzing subjective public narratives for civic mobilization purposes
• Evaluating LLM performance against expert annotators for narrative analysis

Innovation

• Leveraging LLMs to automate narrative annotation
• Evaluating LLM performance against expert annotators
• Extending analysis to political speeches for rhetoric