Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction

2026-04-12

AI Summary

This study investigates whether BERT embeddings implicitly encode multidimensional semantic structures—namely temporal, spatial, causal, and character-related information—in fictional narratives. The authors construct the first token-level narrative annotation dataset across these four dimensions with the aid of large language models and systematically evaluate BERT’s representational capacity using linear probing, class-balanced weighting, confusion matrices, and clustering analysis assessed via Adjusted Rand Index (ARI). Results demonstrate that BERT indeed captures significant narrative information, achieving 94% accuracy and a macro-averaged recall of 0.83 with linear probes. Causal (0.75) and spatial (0.66) dimensions exhibit particularly strong performance, significantly outperforming random baselines. The study also identifies a “boundary leakage” phenomenon, revealing that while narrative semantics are present, they do not form discrete clusters—providing the first token-level evidence of BERT’s implicit encoding of complex narrative structures.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Narrative understanding requires multidimensional semantic structures. This study investigates whether BERT embeddings encode dimensions of fictional narrative semantics -- time, space, causality, and character. Using an LLM to accelerate annotation, we construct a token-level dataset labeled with these four narrative categories plus "others." A linear probe on BERT embeddings (94% accuracy) significantly outperforms a control probe on variance-matched random embeddings (47%), confirming that BERT encodes meaningful narrative information. With balanced class weighting, the probe achieves a macro-average recall of 0.83, with moderate success on rare categories such as causality (recall = 0.75) and space (recall = 0.66). However, confusion matrix analysis reveals "Boundary Leakage," where rare dimensions are systematically misclassified as "others." Clustering analysis shows that unsupervised clustering aligns near-randomly with predefined categories (ARI = 0.081), suggesting that narrative dimensions are encoded but not as discretely separable clusters. Future work includes a POS-only baseline to disentangle syntactic patterns from narrative encoding, expanded datasets, and layer-wise probing.

Problem

• BERT embeddings
• narrative semantics
• time
• space
• causality

Innovation

• BERT embeddings
• narrative semantics
• probing analysis
• token-level annotation
• boundary leakage