Computational frame analysis revisited: On LLMs for studying news coverage

2025-11-21

AI Summary

This study systematically evaluates the effectiveness of generative large language models (LLMs)—including GPT and Claude—for news framing analysis, benchmarking them against bag-of-words models, encoder-only Transformers, and human coding. Using a high-quality, manually annotated dataset comprising six months of U.S. news coverage on the 2022 mpox outbreak, the work adopts methodological pluralism to assess performance across framing identification, classification, and interpretation stages. Results indicate that generative LLMs underperform relative to human coders—and in certain cases, even lag behind smaller, domain-specific models—underscoring the indispensable role of human oversight and validation. The primary contribution is the articulation of a “human–AI staged collaboration” framework for computational framing analysis, which leverages complementary strengths across methods. This paradigm advances reproducible methodology selection and provides a practical roadmap for computational communication research. (149 words)

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Computational approaches have previously shown various promises and pitfalls when it comes to the reliable identification of media frames. Generative LLMs like GPT and Claude are increasingly being used as content analytical tools, but how effective are they for frame analysis? We address this question by systematically evaluating them against their computational predecessors: bag-of-words models and encoder-only transformers; and traditional manual coding procedures. Our analysis rests on a novel gold standard dataset that we inductively and iteratively developed through the study, investigating six months of news coverage of the US Mpox epidemic of 2022. While we discover some potential applications for generative LLMs, we demonstrate that they were consistently outperformed by manual coders, and in some instances, by smaller language models. Some form of human validation was always necessary to determine appropriate model choice. Additionally, by examining how the suitability of various approaches depended on the nature of different tasks that were part of our frame analytical workflow, we provide insights as to how researchers may leverage the complementarity of these approaches to use them in tandem. We conclude by endorsing a methodologically pluralistic approach and put forth a roadmap for computational frame analysis for researchers going forward.

Problem

• Evaluating generative LLMs' effectiveness for media frame analysis
• Comparing LLMs against traditional computational methods and manual coding
• Determining appropriate model selection through human validation processes

Innovation

• Evaluated generative LLMs against traditional methods
• Used novel gold standard dataset for frame analysis
• Proposed combining human validation with model selection