Narrative Shift Detection: A Hybrid Approach of Dynamic Topic Models and Large Language Models

2025-06-25

AI Summary

This paper addresses the challenge of detecting narrative evolution in large-scale media corpora. We propose a hybrid method integrating dynamic topic modeling (DTM) with large language models (LLMs) to distinguish content change from genuine narrative shifts at low cost and high accuracy. Methodologically, we first apply DTM coupled with changepoint detection to identify candidate shift points in text streams; subsequently, we employ an LLM—guided by the Narrative Policy Framework—to perform semantic discrimination and typological classification of identified shifts. Evaluated on a 2009–2023 Wall Street Journal corpus, our framework significantly improves efficiency, scalability, and robustness in pinpointing critical narrative transition points. Its key contribution lies in the first systematic integration of LLM-based semantic understanding into dynamic topic analysis, enabling automated, interpretable, narrative-level parsing. However, fine-grained classification of complex or multi-faceted shift types remains an area for further improvement.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Web Mining and Content Analysis: Models for Web evolution
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

With rapidly evolving media narratives, it has become increasingly critical to not just extract narratives from a given corpus but rather investigate, how they develop over time. While popular narrative extraction methods such as Large Language Models do well in capturing typical narrative elements or even the complex structure of a narrative, applying them to an entire corpus comes with obstacles, such as a high financial or computational cost. We propose a combination of the language understanding capabilities of Large Language Models with the large scale applicability of topic models to dynamically model narrative shifts across time using the Narrative Policy Framework. We apply a topic model and a corresponding change point detection method to find changes that concern a specific topic of interest. Using this model, we filter our corpus for documents that are particularly representative of that change and feed them into a Large Language Model that interprets the change that happened in an automated fashion and distinguishes between content and narrative shifts. We employ our pipeline on a corpus of The Wall Street Journal news paper articles from 2009 to 2023. Our findings indicate that a Large Language Model can efficiently extract a narrative shift if one exists at a given point in time, but does not perform as well when having to decide whether a shift in content or a narrative shift took place.

Problem

• Detect narrative shifts in evolving media over time
• Combine topic models and LLMs for scalable analysis
• Distinguish between content shifts and narrative shifts

Innovation

• Hybrid approach combining topic and language models
• Dynamic narrative shift detection using change points
• Automated content vs narrative shift distinction