More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing

2026-05-12

AI Summary

This work addresses the challenge of catastrophic forgetting and model collapse in large language models during continual editing, which hinders simultaneous factual updating and preservation of general capabilities. The study provides the first theoretical analysis of Lifelong Normalization (LN), revealing that it maintains knowledge consistency through a self-reinforcing stability loop and achieves asymptotically orthogonal, norm-bounded parameter updates via ridge regression. Building on this insight, the authors propose StableEdit, which enhances the stability loop by incorporating explicit warm-up and full-gradient whitening. Experiments demonstrate that StableEdit substantially outperforms existing methods on long-sequence editing tasks, achieving superior editing robustness and stability while maintaining low computational overhead.

Abstract

Lifelong Model Editing aims to continuously update evolving facts in Large Language Models while preserving unrelated knowledge and general capabilities, yet it remains plagued by catastrophic forgetting and model collapse. Empirically, we find that recent editors resilient over long horizons share the same core strategy: Lifelong Normalization (LN), which normalizes value gradients using running statistics. Removing LN causes immediate performance collapse, and we observe a counter-intuitive positive cumulative effect where early edits can promote the success of future edits. Yet the mechanism of LN remains a "black box", leaving its precise role in lifelong stability poorly understood. In this work, we provide the first theoretical account of LN in the lifelong regime. Our analysis reveals a self-reinforcing stability loop and proves that, when combined with ridge-regularized regression, LN yields parameter updates with asymptotic orthogonality and bounded norms, directly mitigating forgetting and systemic collapse. Based on these insights, we derive StableEdit, which strengthens this stability loop via an explicit warm-up stage and full whitening, improving long-horizon stability at minimal overhead. Extensive experiments validate our theory and demonstrate competitive performance. Our code is available at https://github.com/MINE-USTC/StableEdit.

Problem

• Lifelong Model Editing
• Catastrophic Forgetting
• Model Collapse
• Lifelong Normalization
• Stability

Innovation

• Lifelong Normalization
• Model Editing
• Catastrophic Forgetting
• Asymptotic Orthogonality
• StableEdit