Kaczmarz Linear Attention

2026-05-08

AI Summary

This work addresses the high computational cost of Transformer attention in long-context language modeling and the suboptimality of existing linear recurrent models due to fixed update coefficients. It introduces, for the first time, the Kaczmarz projection method into delta-rule sequence modeling, proposing a dynamic step-size mechanism based on an online regression objective with key norm normalization. This approach refines the state update rule of Gated DeltaNet through a single scalar adjustment, without modifying the recurrent architecture or hardware kernels. The method consistently improves accuracy, extrapolation capability, and decoding efficiency. Experiments on a 0.4B-parameter model show a perplexity of 8.09—outperforming Gated DeltaNet’s 8.50—supports context lengths up to 65K tokens, achieves 100% single-query retrieval accuracy, yields a 7.03-point gain in multi-query associative recall, and delivers a 2.1× speedup in decoding throughput at 32K context length.

Abstract

Long-context language modeling remains central to modern sequence modeling, but the quadratic cost of Transformer attention makes scaling computationally prohibitive. Linear recurrent models address this bottleneck by compressing the context into a fixed-size state, making the rule that forgets, writes, and edits information a central design problem. To address state maintenance, Gated DeltaNet (GDN) combines gated state decay with delta-rule residual writes, using a learnable coefficient to balance forgetting and update magnitude. However, this coefficient is learned empirically rather than derived from the underlying objective, which can lead to suboptimal update magnitudes. We revisit the online-regression objective underlying GDN and, inspired by the Kaczmarz projection method, derive the key-norm-normalized dynamic step size $β_t = η_t / (\|k_t\|_2^2 + ε)$ for residual updates. We propose Kaczmarz Linear Attention (KLA), a one-scalar modification of GDN that preserves the state shape, gates, linear recurrence, and chunkwise parallel algorithm. At the 0.4B scale with a 1B-token budget, KLA achieves the lowest validation perplexity among evaluated linear-time baselines, 8.09 versus 8.50 for GDN, and remains stable up to 65K tokens. On controlled tasks, KLA reaches 100% on single-needle-in-a-haystack retrieval, improves 8x multi-query associative recall by 7.03 points over GDN, and delivers 2.1x higher decode throughput at 32K context. These results suggest that the key-norm-normalized Kaczmarz coefficient is a first-order design axis for delta-rule sequence models: it improves accuracy, extrapolation, and decoding efficiency without changing the recurrent state or hardware kernel.

Problem

• linear attention
• long-context modeling
• state update
• delta rule
• Kaczmarz method

Innovation

• Kaczmarz Linear Attention
• linear attention
• delta rule
• online regression
• key-norm normalization