Multiplicative Orthogonal Sequential Editing for Language Models

2026-01-11

AI Summary

This work addresses the numerical instability inherent in existing knowledge editing methods for language models, which rely on additive updates during sequential editing and consequently degrade both editing performance and general model capabilities. To overcome this limitation, the authors propose Multiplicative Orthogonal Sequential Editing (MOSE), a novel paradigm that embeds new knowledge into orthogonal matrices and updates the original parameters multiplicatively. This approach fundamentally avoids the perturbations to matrix condition numbers and norms caused by additive updates, thereby preserving numerical stability. Experimental results demonstrate that MOSE improves sequential editing performance by 12.08% across three large language models while retaining 95.73% of their general downstream task capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Knowledge editing aims to efficiently modify the internal knowledge of large language models (LLMs) without compromising their other capabilities. The prevailing editing paradigm, which appends an update matrix to the original parameter matrix, has been shown by some studies to damage key numerical stability indicators (such as condition number and norm), thereby reducing editing performance and general abilities, especially in sequential editing scenario. Although subsequent methods have made some improvements, they remain within the additive framework and have not fundamentally addressed this limitation. To solve this problem, we analyze it from both statistical and mathematical perspectives and conclude that multiplying the original matrix by an orthogonal matrix does not change the numerical stability of the matrix. Inspired by this, different from the previous additive editing paradigm, a multiplicative editing paradigm termed Multiplicative Orthogonal Sequential Editing (MOSE) is proposed. Specifically, we first derive the matrix update in the multiplicative form, the new knowledge is then incorporated into an orthogonal matrix, which is multiplied by the original parameter matrix. In this way, the numerical stability of the edited matrix is unchanged, thereby maintaining editing performance and general abilities. We compared MOSE with several current knowledge editing methods, systematically evaluating their impact on both editing performance and the general abilities across three different LLMs. Experimental results show that MOSE effectively limits deviations in the edited parameter matrix and maintains its numerical stability. Compared to current methods, MOSE achieves a 12.08% improvement in sequential editing performance, while retaining 95.73% of general abilities across downstream tasks. The code is available at https://github.com/famoustourist/MOSE.

Problem

• knowledge editing
• numerical stability
• sequential editing
• large language models
• parameter matrix

Innovation

• multiplicative editing
• orthogonal matrix
• knowledge editing
• numerical stability
• sequential editing