Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3

2024-06-17

AI Summary

This study addresses speaker attribution for direct speech in literary texts. We systematically evaluate LLaMA-3’s zero-shot and few-shot performance across 28 English novels. To ensure methodological rigor, we introduce a threefold validation framework: (1) controlled ablation experiments to isolate linguistic cues from spurious correlations; (2) memory influence analysis to quantify model reliance on memorized training data; and (3) annotation leakage detection to rule out data contamination. Results demonstrate that LLaMA-3 substantially outperforms both ChatGPT and encoder-based baselines, establishing a new state-of-the-art for this task. Crucially, this work presents the first robust, reproducible evaluation of quotation attribution at scale within authentic literary contexts—free from memory bias and dataset leakage artifacts. All code, annotated data, and experimental protocols are publicly released, providing a benchmark resource and methodological blueprint for literary NLP.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have shown promising results in a variety of literary tasks, often using complex memorized details of narration and fictional characters. In this work, we evaluate the ability of Llama-3 at attributing utterances of direct-speech to their speaker in novels. The LLM shows impressive results on a corpus of 28 novels, surpassing published results with ChatGPT and encoder-based baselines by a large margin. We then validate these results by assessing the impact of book memorization and annotation contamination. We found that these types of memorization do not explain the large performance gain, making Llama-3 the new state-of-the-art for quotation attribution in English literature. We release publicly our code and data.

Problem

• Large Language Models
• Speaker Identification
• Literary Texts

Innovation

• LLaMa3 model
• speaker identification
• English literature