UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Large Language Models

2025-05-20

AI Summary

To address the challenges of efficiency, generalization, training-free operation, topic-agnosticism, and external memory independence in lifelong knowledge editing for large language models (LLMs), this paper proposes the first triple-free editing paradigm—free of training, subject dependency, and external memory. Our method computes parameter offsets via lightweight linear algebra operations, introduces a lifelong normalization strategy to mitigate distribution shift, and enables end-to-end gradient-free online editing through dynamic feature statistics. Evaluated on UltraEditBench (2M samples), it sustains high accuracy across million-scale sequential edits. It achieves >7× faster editing speed and <1/3 GPU memory consumption compared to state-of-the-art methods. Notably, it is the first to perform full 7B-model editing on a single 24GB consumer-grade GPU—demonstrating unprecedented practicality for resource-constrained lifelong editing scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Lifelong learning enables large language models (LLMs) to adapt to evolving information by continually updating their internal knowledge. An ideal system should support efficient, wide-ranging updates while preserving existing capabilities and ensuring reliable deployment. Model editing stands out as a promising solution for this goal, offering a focused and efficient way to revise a model's internal knowledge. Although recent paradigms have made notable progress, they often struggle to meet the demands of practical lifelong adaptation at scale. To bridge this gap, we propose ULTRAEDIT-a fundamentally new editing solution that is training-, subject- and memory-free, making it particularly well-suited for ultra-scalable, real-world lifelong model editing. ULTRAEDIT performs editing through a self-contained process that relies solely on lightweight linear algebra operations to compute parameter shifts, enabling fast and consistent parameter modifications with minimal overhead. To improve scalability in lifelong settings, ULTRAEDIT employs a lifelong normalization strategy that continuously updates feature statistics across turns, allowing it to adapt to distributional shifts and maintain consistency over time. ULTRAEDIT achieves editing speeds over 7x faster than the previous state-of-the-art method-which was also the fastest known approach-while consuming less than 1/3 the VRAM, making it the only method currently capable of editing a 7B LLM on a 24GB consumer-grade GPU. Furthermore, we construct ULTRAEDITBENCH-the largest dataset in the field to date, with over 2M editing pairs-and demonstrate that our method supports up to 1M edits while maintaining high accuracy. Comprehensive experiments on four datasets and six models show that ULTRAEDIT consistently achieves superior performance across diverse model editing scenarios. Our code is available at: https://github.com/XiaojieGu/UltraEdit.

Problem

• Enables lifelong learning in LLMs without training or memory constraints
• Achieves fast, scalable editing via lightweight linear algebra operations
• Maintains model accuracy across 1M+ edits with minimal resource usage

Innovation

• Training-, subject-, and memory-free lifelong editing solution
• Self-contained process with lightweight linear algebra operations
• Lifelong normalization strategy for scalability and consistency