GeoEdit: Geometric Knowledge Editing for Large Language Models

2025-02-27

AI Summary

To address the fundamental trade-off between precise injection of new knowledge and preservation of general capabilities during dynamic knowledge updating in large language models (LLMs), this paper proposes a parameter-geometry-based knowledge editing framework. Methodologically, it introduces: (1) a novel direction-aware knowledge identification mechanism to distinguish neurons encoding new versus general knowledge; (2) a “forget–relearn” reverse-editing paradigm; and (3) an importance-guided task vector fusion strategy enabling neuron-level adaptive weighting. Technically, the approach integrates geometric directional analysis, orthogonality-constrained optimization, and task vector decomposition/fusion. Evaluated on two public benchmarks, the framework significantly outperforms state-of-the-art methods across key dimensions—editing accuracy, generalization retention, and multi-hop knowledge consistency—demonstrating robust performance while maintaining model-wide functionality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs). Consequently, various model editing methods have been developed to update specific knowledge within LLMs. However, training-based approaches often struggle to effectively incorporate new knowledge while preserving unrelated general knowledge. To address this challenge, we propose a novel framework called Geometric Knowledge Editing (GeoEdit). GeoEdit utilizes the geometric relationships of parameter updates from fine-tuning to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. By employing a direction-aware knowledge identification method, we avoid updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model's generalization ability. For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a"forget-then-learn"editing strategy for opposite directions. Additionally, we introduce an importance-guided task vector fusion technique that filters out redundant information and provides adaptive neuron-level weighting, further enhancing model editing performance. Extensive experiments on two publicly available datasets demonstrate the superiority of GeoEdit over existing state-of-the-art methods.

Problem

• Updates specific knowledge in large language models
• Preserves unrelated general knowledge during updates
• Enhances model editing performance through geometric relationships

Innovation

• GeoEdit uses geometric knowledge editing.
• Direction-aware identification preserves generalization.
• Importance-guided task vector enhances editing.