Identifying economic narratives in large text corpora -- An integrated approach using Large Language Models

2025-06-18

AI Summary

Economic narrative identification suffers from inadequate deep semantic understanding and ambiguous task boundaries—particularly with semantic role labeling—hindering reliable automated extraction. Method: We propose an end-to-end, LLM-based structured extraction framework, grounded in a rigorously defined economic narrative schema. For the first time, we systematically evaluate GPT-4o’s narrative extraction performance on inflation-related articles from *The Wall Street Journal* and *The New York Times*, benchmarking outputs against expert annotations. Results: GPT-4o consistently generates syntactically and semantically compliant structured narratives, achieving near-expert accuracy on standard texts; however, it exhibits substantial performance degradation on documents featuring implicit causality, high conceptual density, or multi-event entanglement. This work not only demonstrates the feasibility of deploying LLMs as lightweight narrative extraction tools but also establishes a reproducible evaluation paradigm and methodological framework for LLM applications in the social sciences.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Large Multimodal Models (LMMs)
• Game Theory and Economic Paradigms: Applications

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

As interest in economic narratives has grown in recent years, so has the number of pipelines dedicated to extracting such narratives from texts. Pipelines often employ a mix of state-of-the-art natural language processing techniques, such as BERT, to tackle this task. While effective on foundational linguistic operations essential for narrative extraction, such models lack the deeper semantic understanding required to distinguish extracting economic narratives from merely conducting classic tasks like Semantic Role Labeling. Instead of relying on complex model pipelines, we evaluate the benefits of Large Language Models (LLMs) by analyzing a corpus of Wall Street Journal and New York Times newspaper articles about inflation. We apply a rigorous narrative definition and compare GPT-4o outputs to gold-standard narratives produced by expert annotators. Our results suggests that GPT-4o is capable of extracting valid economic narratives in a structured format, but still falls short of expert-level performance when handling complex documents and narratives. Given the novelty of LLMs in economic research, we also provide guidance for future work in economics and the social sciences that employs LLMs to pursue similar objectives.

Problem

• Extracting economic narratives from large text corpora
• Comparing LLM performance with expert annotations
• Guidance for using LLMs in economic research

Innovation

• Uses GPT-4o for economic narrative extraction
• Compares GPT-4o outputs with expert annotations
• Provides guidance for LLMs in economic research