Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling

2026-07-14

AI Summary

This work addresses the unreliability of language model outputs when encoded parametric knowledge becomes outdated, incomplete, or conflicts with contextual information. To mitigate overreliance on such internalized knowledge, the paper proposes a novel paradigm—Knowledge-Lite Language Models (KLLMs)—which systematically anonymizes named entities during pretraining to reduce the model’s dependence on memorized facts and encourage reasoning grounded in contextual evidence. This approach requires only modifications to the pretraining corpus and is compatible across model scales. Experimental results demonstrate that KLLMs significantly outperform baseline models on contextual question answering, fact verification, and hallucination detection tasks, achieving performance gains of 20–25% in scenarios with incomplete retrieval evidence. Moreover, KLLMs exhibit better calibration and more reliable abstention behavior when answers cannot be confidently inferred from context.

Abstract

Language models encode substantial factual knowledge in their parameters, which can lead to unreliable behavior when this knowledge is outdated, incomplete, or misaligned with the provided context. In this work, we study whether modifying the pretraining signal can systematically shift models away from parametric recall and toward evidence-grounded reasoning. We introduce Knowledge--''Less'' Language Models (KLLMs), a fundamentally different epistemic training paradigm for LLMs, which are pretrained on corpora in which named entities are anonymized, thereby removing a primary channel for entity-linked factual supervision. This intervention substantially reduces closed-book factual recall, while often improving performance on tasks where relevant information is provided as context. Across multiple model scales, KLLMs consistently outperform matched baselines on contextual question answering, fact verification, and hallucination detection benchmarks. Crucially, in retrieval-grounded settings with imperfect evidence, KLLMs show improved robustness and achieve up to 20--25\% relative gains over standard language models. They further exhibit better calibration, with improved ECE, Brier score, and AUROC, as well as more reliable abstention behavior. Our results demonstrate that suppressing entity-linked supervision during pretraining induces a shift in epistemic behavior: KLLMs rely less on parametric knowledge and more on external evidence, leading to improved reliability under realistic conditions. This suggests that pretraining-time control over knowledge acquisition can complement retrieval-augmented and tool-based systems by providing a more evidence-sensitive base model.

Problem

• parametric recall
• evidence-grounded reasoning
• factual knowledge
• hallucination
• language models

Innovation

• knowledge suppression
• evidence-grounded reasoning
• entity anonymization
• parametric recall
• language model calibration