Uncovering Context Reliance in Unstructured Knowledge Editing

2026-02-21

AI Summary

This work identifies and theoretically demonstrates that large language models exhibit an inherent reliance on original contextual cues during knowledge editing—a fundamental limitation under gradient-based optimization—which impairs their ability to accurately retrieve edited knowledge when context is absent. To address this issue, the authors propose COIN, a context-independent editing framework that steers the model to focus on localized knowledge representations rather than contextual patterns, thereby substantially enhancing editing robustness. Experimental results show that COIN reduces context dependence by 45.2% and improves editing success rates by 23.6% over strong baselines, establishing a new paradigm for more reliable and context-agnostic knowledge updates in large language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Editing Large language models (LLMs) with real-world, unstructured knowledge is essential for correcting and updating their internal parametric knowledge. In this work, we revisit the fundamental next-token prediction (NTP) as a candidate paradigm for unstructured editing. We identify Context Reliance as a critical failure mode of NTP-based approaches, where knowledge acquired from edited text becomes highly dependent on its preceding context, leading to recall failures when that context is absent during inference. This hypothesis is supported by our empirical validation that prepending context during inference recovers knowledge recall. We further theoretically demonstrate that Context Reliance is an inherent consequence of gradient-based optimization, which tends to bind acquired knowledge to a specific aggregated contextual representation. To address this, we propose a simple yet effective COntext-INdependent editing framework (COIN), encouraging model to focus on knowledge within local scope rather than memorizing contextual patterns. Evaluations show that COIN reduces Context Reliance by 45.2% and outperforms strong baselines by 23.6% in editing success rate, highlighting the vital role of mitigating Context Reliance for robust editing.

Problem

• Context Reliance
• Unstructured Knowledge Editing
• Large Language Models
• Next-Token Prediction
• Knowledge Recall

Innovation

• Context Reliance
• Knowledge Editing
• Large Language Models
• Next-Token Prediction
• COIN