Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms

2026-04-23

AI Summary

Existing evaluation methods struggle to distinguish between large language models’ factual recall and their reliance on the surface forms of entities, often misjudging their non-literal memorization capabilities. This work introduces RedirectQA, a novel dataset that leverages Wikipedia redirect information to associate Wikidata triples with diverse surface variants—including aliases, abbreviations, spelling variations, and common errors—enabling systematic assessment of model consistency in factual recall across entity name variations. Using this dataset, combined with frequency analysis and surface-conditioned question-answering experiments across 13 large language models, we find that models exhibit robustness to minor orthographic changes but are sensitive to lexical-level variations such as aliases and abbreviations. Furthermore, both entity frequency and surface-form frequency influence accuracy, with entity frequency contributing independently to performance.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Understanding what kinds of factual knowledge large language models (LLMs) memorize is essential for evaluating their reliability and limitations. Entity-based QA is a common framework for analyzing non-verbatim memorization, but typical evaluations query each entity using a single canonical surface form, making it difficult to disentangle fact memorization from access through a particular name. We introduce RedirectQA, an entity-based QA dataset that uses Wikipedia redirect information to associate Wikidata factual triples with categorized surface forms for each entity, including alternative names, abbreviations, spelling variants, and common erroneous forms. Across 13 LLMs, we examine surface-conditioned factual memorization and find that prediction outcomes often change when only the entity surface form changes. This inconsistency is category-dependent: models are more robust to minor orthographic variations than to larger lexical variations such as aliases and abbreviations. Frequency analyses further suggest that both entity- and surface-level frequencies are associated with accuracy, and that entity frequency often contributes beyond surface frequency. Overall, factual memorization appears neither purely surface-specific nor fully surface-invariant, highlighting the importance of surface-form diversity in evaluating non-verbatim memorization.

Problem

• non-verbatim memorization
• entity surface forms
• large language models
• factual knowledge
• surface-form diversity

Innovation

• non-verbatim memorization
• entity surface forms
• RedirectQA
• large language models
• factual knowledge