CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners

2025-03-20

AI Summary

Existing knowledge editing methods struggle to generalize from isolated fact updates to complex downstream tasks—such as multi-hop reasoning—that depend on the modified knowledge. This paper proposes a circuit-aware knowledge editing framework, the first to integrate neural circuit analysis into knowledge editing. By identifying and jointly editing cross-layer reasoning pathways, it shifts the paradigm from single-layer local parameter modification to multi-layer reasoning-path reconstruction. The method comprises three components: (1) circuit-guided construction of editing examples, (2) multi-layer joint parameter optimization, and (3) supervision targeting reasoning-path activation during inference. Evaluated on the MQuAKE benchmark, our approach improves multi-hop reasoning accuracy by an average of 20%. Moreover, edited knowledge demonstrates significantly enhanced invocation accuracy and stability across related tasks, validating its transferability and consistency.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Acquisition
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Knowledge Editing (KE) enables the modification of outdated or incorrect information in large language models (LLMs). While existing KE methods can update isolated facts, they struggle to generalize these updates to multi-hop reasoning tasks that depend on the modified knowledge. Through an analysis of reasoning circuits -- the neural pathways LLMs use for knowledge-based inference, we observe that current layer-localized KE approaches, such as MEMIT and WISE, which edit only single or a few model layers, struggle to effectively incorporate updated information into these reasoning pathways. To address this limitation, we propose CaKE (Circuit-aware Knowledge Editing), a novel method that enables more effective integration of updated knowledge in LLMs. CaKE leverages strategically curated data, guided by our circuits-based analysis, that enforces the model to utilize the modified knowledge, stimulating the model to develop appropriate reasoning circuits for newly integrated knowledge. Experimental results show that CaKE enables more accurate and consistent use of updated knowledge across related reasoning tasks, leading to an average of 20% improvement in multi-hop reasoning accuracy on MQuAKE dataset compared to existing KE methods. We release the code and data in https://github.com/zjunlp/CaKE.

Problem

• Improves generalization of knowledge updates in LLMs
• Addresses limitations of layer-localized knowledge editing methods
• Enhances multi-hop reasoning accuracy with updated knowledge

Innovation

• Circuit-aware Knowledge Editing (CaKE) method
• Strategic data curation for knowledge integration
• Improved multi-hop reasoning accuracy by 20%