Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

2026-06-21

AI Summary

This work addresses the degradation in editing accuracy caused by entangled semantic representations in batch knowledge editing. To this end, the authors propose an orthogonal representation editing mechanism that constructs a universal semantic subspace within the hidden layers of large language models. By imposing orthogonality constraints between editing vectors and employing gated nonlinear representation heads, the method achieves semantic disentanglement and enables precise knowledge injection. Furthermore, it supports adaptive learning of editing locations across multilingual scenarios. Extensive experiments demonstrate that the proposed approach significantly outperforms existing techniques, exhibiting particularly strong performance in cross-lingual knowledge editing tasks.

Abstract

Knowledge editing aims to efficiently update factual information in Large Language Models (LLMs) without full retraining. However, existing methods still suffer from performance degradation in batch knowledge editing. We identify that semantic representation entanglement, such as overlapping concepts and shared syntactic patterns, accumulates interference in the representation space and reduces editing precision. To bridge this gap, in this paper, we propose Orthogonal Representation Editing (ORE), which performs edits in the hidden representation space of LLMs by constructing a general semantic subspace and enforcing orthogonal constraints on edit vectors, effectively decoupling semantic entanglement. Furthermore, we introduce a gated non-linear representation head to enable adaptive learning of editing locations and precise control over knowledge injection. Extensive experiments show that ORE outperforms existing methods and achieves superior performance in cross-lingual knowledge editing scenarios. We release our code at https://github.com/YVVH/ORE.

Problem

• knowledge editing
• semantic entanglement
• batch editing
• representation space
• Large Language Models

Innovation

• Orthogonal Representation Editing
• Semantic Disentanglement
• Batch Knowledge Editing
• Representation Space
• Large Language Models