The Query Knows What to Forget: A Second Erase Direction for Linear Attention

2026-08-13

AI Summary

This study addresses the state interference issue inherent in linear attention during long-context retrieval by proposing a query-derived erasure direction mechanism. Specifically, the method constructs a secondary erasure vector orthogonal to the key space to eliminate residual noise from outdated states, while integrating a fast weight editing strategy to optimize memory updates. Experimental results demonstrate that this approach effectively mitigates retrieval performance degradation, doubling the effective context length on the s-niah-1 benchmark. By providing an efficient solution for enhancing long-range dependency modeling in linear attention architectures, this work significantly improves processing efficacy for long-text tasks.

Abstract

Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token. However, the interference in its reads is measured through the query, and the erase step cannot reach it. We introduce the Query-derived Erase Direction (QED). QED adds a second erase direction derived from the query and orthogonal to the key. In the fast-weight view, a key-directed delta edit cannot change the key-orthogonal part of a read. It uses the editable part to cancel old-state content measured along the query. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.

Problem

• Linear Attention
• State Interference
• Long Context Retrieval
• Delta Rule Models

Innovation

• Query-derived Erase Direction
• Linear Attention
• Gated DeltaNet-2
• Orthogonal Erase
• Long Context Retrieval