AnyEdit: Edit Any Knowledge Encoded in Language Models

2025-02-08

AI Summary

Large language models (LLMs) face significant challenges in precisely editing long-form, multimodal knowledge—such as poetry, code, and mathematical derivations—due to the “effectiveness bottleneck” inherent in existing single-token editing approaches. This work introduces the first autoregressive editing paradigm that overcomes this limitation, grounded in sequence chunking and iterative critical-token editing, theoretically justified by the chain rule of mutual information. The method enables end-to-end consistent updates for knowledge of arbitrary length and format. It comprises three core components: a knowledge chunking strategy, a critical-token localization mechanism, and mutual-information-driven editing modeling. Evaluated on UnKEBench, AKEW, and our newly introduced EditEverything benchmark, the approach achieves an average 21.5% improvement in editing accuracy. Moreover, it functions as a plug-and-play module, substantially expanding the capability frontier of existing editing methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical derivations. These limitations arise from their reliance on editing a single token's hidden state, a limitation we term"efficacy barrier". To solve this, we propose AnyEdit, a new autoregressive editing paradigm. It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual Information, showing its ability to update any knowledge within LLMs. Empirically, it outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge. Additionally, AnyEdit serves as a plug-and-play framework, enabling current editing methods to update knowledge with arbitrary length and format, significantly advancing the scope and practicality of LLM knowledge editing.

Problem

• Efficient and precise knowledge updates in LLMs
• Handling long-form knowledge in diverse formats
• Overcoming efficacy barrier in model editing methods

Innovation

• Autoregressive editing paradigm
• Sequential chunk decomposition
• Plug-and-play framework