Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

2026-08-12

AI Summary

This study investigates how extended context lengths during training influence the trade-off between parametric knowledge and context dependence in large language models. Through a combination of pretraining and supervised fine-tuning experiments, gradient analysis, causal interventions, and module importance assessments, the authors uncover an “information abundance paradox”: while longer contexts improve performance on certain tasks, they simultaneously impair the model’s ability to internalize knowledge into its parameters, thereby reducing robustness in settings with no context or misleading context. The work identifies an optimal training context length beyond which performance on closed-book question answering significantly degrades, accompanied by an increased reliance on attention mechanisms rather than feedforward networks for knowledge retention.

Abstract

Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that training on longer contexts will only help the model by exposing it to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization. We propose the Information Abundance Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically, thereby increasing reliance on context. In pretraining with long documents, increasing the context window improves language modeling, natural language understanding, and closed-book MCQA only up to an intermediate optimum, after which performance consistently declines. In supervised fine-tuning, more task-relevant train-time context improves performance with supporting context, but reduces robustness when context is absent or misleading at test time. Our analysis suggests that this behavior arises when longer context provides a lower complexity solution. Mechanistically, training with informative context shifts gradient pressure from feed-forward networks, often linked to parametric knowledge, toward attention modules, and causal interventions show that this shift increases reliance on context during inference. Overall, these findings support the Information Abundance Paradox and suggest that scaling toward near-infinite context is not simply a matter of supplying more data, even when high-quality long-context data is abundant.

Problem

• Information Abundance Paradox
• long-context training
• parametric knowledge
• contextualization
• language models

Innovation

• Information Abundance Paradox
• parametric knowledge
• long-context training
• contextualization
• gradient shift