From Backward Spreading to Forward Replay: Revisiting Target Construction in LLM Parameter Editing

2026-04-30

AI Summary

Existing parameter editing methods for large language models rely on backpropagation to construct target hidden states, lacking theoretical grounding and suffering from ambiguous capability boundaries and potential failure modes. This work proposes a novel paradigm that anchors the first edited layer and generates multi-layer compatible target hidden states through forward replay. For the first time, it replaces backward diffusion with purely forward propagation, significantly improving the accuracy and consistency of target hidden states across edited layers—without increasing computational complexity or altering the overall editing pipeline. The proposed framework seamlessly integrates into mainstream parameter editing approaches, effectively enhancing their editing performance.

Abstract

LLM parameter editing methods commonly rely on computing an ideal target hidden-state at a target layer (referred as anchor point) and distributing the target vector to multiple preceding layers (commonly known as backward spreading) for cooperative editing. Although widely used for a long time, its underlying basis have not been systematically investigated. In this paper, we first conduct a systematic study of its foundations, which helps clarify its capability boundaries, practical considerations, and potential failure modes. Then, we propose a simple and elegant alternative that replaces backward spreading with forward-propagation. Instead of optimizing the target at the last editing layer, we optimize the anchor point at the first editing layer, and then propagate it forward to obtain accurate and mutually compatible target hidden-states for all subsequent editing layers. This approach achieves the same computational complexity as existing methods while producing more accurate layer-wise targets. Our method is simple, without interfering with either the computation of the initial target hidden state or any other components of the subsequent editing pipeline, and thus constituting a benefit for a wide range of LLM parameter editing methods.

Problem

• LLM parameter editing
• backward spreading
• target construction
• hidden-state propagation
• model editing

Innovation

• forward replay
• parameter editing
• target construction
• backward spreading
• large language models