Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression

2026-04-21

AI Summary

This work addresses the challenge of catastrophic forgetting and prohibitive computational costs associated with full fine-tuning when continuously updating knowledge in large language models. To this end, the authors propose LightEdit, a novel framework that integrates retrieval-augmented knowledge selection with a decoding-stage mechanism that suppresses the probability of outdated factual knowledge. Without requiring extensive retraining, LightEdit enables efficient and stable lifelong knowledge editing. Empirical evaluations demonstrate that the method substantially improves editing accuracy and cross-dataset generalization on benchmarks such as ZSRE, CounterFact, and RIPE, while significantly reducing computational overhead. This approach establishes a scalable and cost-effective paradigm for dynamic knowledge updating in large language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) require frequent knowledge updates to reflect changing facts and mitigate hallucinations. To meet this demand, lifelong knowledge editing has emerged as a continual approach to modify specific pieces of knowledge without retraining the entire model. Existing parameter editing methods struggle with stability during sequential edits due to catastrophic forgetting. While retrieval-based approaches are proposed to alleviate this issue, their applicability remains limited across various datasets because of high training costs. To address these limitations and enhance scalability in lifelong settings, we propose LightEdit. Our framework first selects relevant knowledge from retrieved information to modify the query effectively. It then incorporates a decoding strategy to suppress the model's original knowledge probabilities, thereby enabling efficient edits based on the selected information. Extensive experiments on ZSRE, Counterfact, and RIPE benchmarks demonstrate that LightEdit outperforms existing lifelong knowledge editing methods. Furthermore, by minimizing training costs, LightEdit achieves cost-effective scalability, enabling easy adaptation to various datasets.

Problem

• lifelong knowledge editing
• catastrophic forgetting
• scalability
• knowledge updating
• large language models

Innovation

• lifelong knowledge editing
• selective knowledge suppression
• retrieval-based editing
• scalable LLM updating
• decoding strategy