NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database

2025-07-23

AI Summary

To address the challenges of catastrophic forgetting, degradation of general capabilities, and poor scalability in knowledge editing for large language models (LLMs), this paper proposes a knowledge editing framework based on a neural key-value (KV) database. Methodologically, editing is formulated as a conditional KV lookup, augmented with a nonlinear gated retrieval mechanism that precisely activates target parameters while suppressing interference during editing—enabling simultaneous updates of up to 100K factual assertions. Experiments on GPT2-XL, GPT-J, and Llama-3 demonstrate substantial improvements in editing accuracy on ZsRE and CounterFacts benchmarks, supporting edits of 100K facts—50× larger than prior state-of-the-art methods—while preserving original model performance across six downstream tasks. This work achieves, for the first time, scalable, high-fidelity, and robust knowledge editing in LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Efficiently editing knowledge stored in large language models (LLMs) enables model updates without large-scale training. One possible solution is Locate-and-Edit (L&E), allowing simultaneous modifications of a massive number of facts. However, such editing may compromise the general abilities of LLMs and even result in forgetting edited facts when scaling up to thousands of edits. In this paper, we model existing linear L&E methods as querying a Key-Value (KV) database. From this perspective, we then propose NeuralDB, an editing framework that explicitly represents the edited facts as a neural KV database equipped with a non-linear gated retrieval module, % In particular, our gated module only operates when inference involves the edited facts, effectively preserving the general abilities of LLMs. Comprehensive experiments involving the editing of 10,000 facts were conducted on the ZsRE and CounterFacts datasets, using GPT2-XL, GPT-J (6B) and Llama-3 (8B). The results demonstrate that NeuralDB not only excels in editing efficacy, generalization, specificity, fluency, and consistency, but also preserves overall performance across six representative text understanding and generation tasks. Further experiments indicate that NeuralDB maintains its effectiveness even when scaled to 100,000 facts ( extbf{50x} more than in prior work).

Problem

• Efficiently edit knowledge in LLMs without retraining
• Prevent performance loss when scaling to thousands of edits
• Maintain LLM general abilities during large-scale fact updates

Innovation

• Neural KV database for knowledge editing
• Non-linear gated retrieval module
• Scales to 100,000 facts efficiently