HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing

2026-05-02

AI Summary

This work addresses the challenges of knowledge obsolescence and high editing costs in deployed large language models, where existing methods struggle to maintain both accuracy and knowledge retention under massive sequential edits. The authors propose HoReN, a parameter-efficient editing framework that innovatively integrates MLP layers with a discrete key-value codebook, treating codebook entries as knowledge keys and Hopfield memory patterns. To enhance robustness, HoReN employs unit hyperspherical projection to eliminate magnitude interference and incorporates damped attractor dynamics for stable querying. Evaluated on benchmarks including ZsRE, WikiBigEdit, and UnKE, HoReN significantly outperforms current approaches, supporting up to 50,000 consecutive edits while consistently maintaining overall performance above 0.9.

Abstract

Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment, yet retraining is costly prohibitive, motivating model editing in lifelong settings that updates targeted behavior without harming the rest of the model. One line of work installs new facts by directly modifying base weights through locate-then-edit procedures, but accumulated edits progressively disrupt originally preserved knowledge, even with constraint-based projections. A complementary line leaves base weights intact and routes edits through external memory, but it faces routing challenges and its performance degrades at scale. We propose HoReN, a codebook-based parameter-preserving editor with enhanced routing built on three ideas. First, HoReN wraps a single MLP layer with a discrete key-value codebook, where each entry is interpreted simultaneously as a knowledge-memory key and a modern Hopfield stored pattern. Second, both keys and queries are projected onto the unit hypersphere so retrieval is governed by angular similarity, removing magnitude-driven mismatches between an edit prompt and its rephrasings. Third, the query is refined through damped Hopfield attractor dynamics, so paraphrases relax into the correct stored pattern's basin of attraction while unrelated queries remain undisturbed. HoReN achieves well-edited performance with consistent gains across diverse benchmarks spanning standard ZsRE, structured WikiBigEdit, and unstructured UnKE evaluations. Moreover, HoReN scales to 50K sequential edits on ZsRE with stable overall performance above 0.9, while prior editors collapse or degrade severely before reaching 10K. Our code is available at https://github.com/ha11ucin8/HoReN.

Problem

• model editing
• large language models
• knowledge updating
• lifelong learning
• sequential editing

Innovation

• Hopfield retrieval
• model editing
• codebook-based memory
• angular similarity
• attractor dynamics