Conflict-Resolving and Sharpness-Aware Minimization for Generalized Knowledge Editing with Multiple Updates

2026-02-03

AI Summary

This work addresses the challenges of weak generalization, poor stability, and interference between old and new knowledge that commonly arise during multi-round knowledge updates in large language models. To this end, the authors propose CoRSA, a novel framework that uniquely integrates sharpness-aware minimization with knowledge conflict resolution through parameter-efficient fine-tuning, achieving a balance between high generalization and low forgetting. Specifically, CoRSA enhances training stability by minimizing loss curvature and mitigates knowledge interference by maximizing the representational separation between old and new knowledge. Experimental results demonstrate that CoRSA outperforms LoRA by an average of 12.42% across three factual editing benchmarks, reduces forgetting by 27.82% under multi-round updates, and improves Pass@5 by 5.48% on code generation tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models (LLMs) rely on internal knowledge to solve many downstream tasks, making it crucial to keep them up to date. Since full retraining is expensive, prior work has explored efficient alternatives such as model editing and parameter-efficient fine-tuning. However, these approaches often break down in practice due to poor generalization across inputs, limited stability, and knowledge conflict. To address these limitations, we propose the CoRSA (Conflict-Resolving and Sharpness-Aware Minimization) training framework, a parameter-efficient, holistic approach for knowledge editing with multiple updates. CoRSA tackles multiple challenges simultaneously: it improves generalization to different input forms and enhances stability across multiple updates by minimizing loss curvature, and resolves conflicts by maximizing the margin between new and prior knowledge. Across three widely used fact editing benchmarks, CoRSA achieves significant gains in generalization, outperforming baselines with average absolute improvements of 12.42% over LoRA and 10% over model editing methods. With multiple updates, it maintains high update efficacy while reducing catastrophic forgetting by 27.82% compared to LoRA. CoRSA also generalizes to the code domain, outperforming the strongest baseline by 5.48% Pass@5 in update efficacy.

Problem

• knowledge editing
• large language models
• generalization
• stability
• knowledge conflict

Innovation

• knowledge editing
• sharpness-aware minimization
• conflict resolution
• parameter-efficient fine-tuning
• catastrophic forgetting