From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation

2026-03-02

AI Summary

This study addresses the lack of high-quality, structured annotation methods for narrative structures of economic events in news discourse, which hinders natural language processing models from effectively capturing complex causal relationships. To bridge this gap, the authors propose a directed acyclic graph (DAG)-based narrative annotation framework that integrates qualitative content analysis, where nodes represent events and directed edges encode causal relations. The framework incorporates local structural constraints—such as one-hop neighborhood restrictions—to reduce inter-annotator variability. Through a 6×3 factorial experiment, the authors systematically evaluate how different graph representations and distance metrics affect annotation reliability, finding that lenient metrics tend to overestimate agreement, whereas local constraints significantly improve Krippendorff’s α. The project publicly releases an implementation of α adapted for graph-structured data, offering a robust methodological foundation and practical guidance for narrative graph annotation.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Machine Learning: Graph-based Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Bridging structured and unstructured data
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Narratives in news discourse play a critical role in shaping public understanding of economic events, such as inflation. Annotating and evaluating these narratives in a structured manner remains a key challenge for Natural Language Processing (NLP). In this work, we introduce a narrative graph annotation framework that integrates principles from qualitative content analysis (QCA) to prioritize annotation quality by reducing annotation errors. We present a dataset of inflation narratives annotated as directed acyclic graphs (DAGs), where nodes represent events and edges encode causal relations. To evaluate annotation quality, we employed a $6\times3$ factorial experimental design to examine the effects of narrative representation (six levels) and distance metric type (three levels) on inter-annotator agreement (Krippendorrf's $α$), capturing the presence of human label variation (HLV) in narrative interpretations. Our analysis shows that (1) lenient metrics (overlap-based distance) overestimate reliability, and (2) locally-constrained representations (e.g., one-hop neighbors) reduce annotation variability. Our annotation and implementation of graph-based Krippendorrf's $α$ are open-sourced. The annotation framework and evaluation results provide practical guidance for NLP research on graph-based narrative annotation under HLV.

Problem

• narrative annotation
• inter-annotator agreement
• human label variation
• graph-based representation
• qualitative content analysis

Innovation

• narrative graph annotation
• qualitative content analysis
• inter-annotator agreement
• Krippendorff's alpha
• human label variation