Characterizing Narrative Content in Web-scale LLM Pretraining Data

2026-06-17

AI Summary

This study addresses the lack of systematic characterization of narrative content in current large-scale language model pretraining corpora. Drawing on narrative theory, the authors propose an 11-dimensional analytical framework encompassing agency, setting, and events. They fine-tune a RoBERTa model (dubbed NarraBERT) using human annotations on 400 text excerpts and perform large-scale inference over three million web-collected passages to construct NarraDolma—the first fine-grained dataset of narrative structures. This work enables, for the first time, quantitative measurement of multidimensional narrative features across trillion-token pretraining corpora, revealing their uneven distribution across sources and topics. The findings demonstrate that narrative traits can be effectively modeled and highlight that prevailing data curation strategies overlook such structural disparities. The model and dataset are publicly released.

Abstract

The narrative composition of web-scale LLM pretraining corpora remains largely unexplored even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After sampling and annotating a diverse set of 400 passages, we finetune and validate NarraBERT, a RoBERTa-based model for fine-grained narrative prediction. We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma. We find (i) narrative structure is measurable at scale across extremely heterogeneous data, (ii) we uncover a continuous, multidimensional narrative structure underlying web text, and (iii) narrative qualities are unequally distributed across pretraining sources and topics in ways that current curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data and for studying how data composition affects narrative reasoning tasks. We publicly release NarraDolma and NarraBERT.

Problem

• narrative content
• LLM pretraining data
• web-scale corpora
• narrative structure
• data composition

Innovation

• narrative analysis
• pretraining data characterization
• NarraBERT
• NarraDolma
• multidimensional narrative structure