Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

2026-07-08

AI Summary

This work addresses the quadratic computational complexity of standard self-attention in long sequences, which hinders model scalability. The study systematically compares Softmax attention with four recurrent linear attention architectures from the DeltaNet family, unifying their memory mechanisms under a common formulation, and introduces a lightweight Cross-Layer Value Routing (CLVR) mechanism. Experiments at a scale of 350M parameters and 15B tokens reveal that Kimi Delta Attention paired with the Muon optimizer achieves the lowest validation loss, while a pure Gated DeltaNet stack attains the highest training throughput under AdamW. CLVR consistently reduces validation loss across both DeltaNet and its gated variants. This is the first systematic analysis comparing multiple linear attention variants in terms of representational capacity, memory decay characteristics, and training efficiency.

Abstract

Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. We express these mechanisms in a common recurrent-memory notation, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity. Our experiments center on 350M-parameter models trained for 15B tokens, and include optimizer and learning-rate comparisons, hybrid-versus-pure stack comparisons, sequence-length runtime measurements, larger DeltaNet runs at 1.3B and 3B parameters, and a small set of downstream evaluations. The reported speed results measure training throughput and iteration time; we do not provide an empirical inference-speed benchmark. Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, a pure Gated DeltaNet stack trained with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW in the matched architecture settings we evaluate. We introduce and evaluate lightweight cross-layer routing mechanisms for DeltaNet-style memories. The most natural DeltaNet-inspired formulation, forwarding a lower layer's delta-rule write error into the next layer's value target, does not improve over matched baselines. Routing into the aligned hidden stream and forwarding the write value instead yields a modest improvement in the matched runs we report: Cross-Layer Value Routing (CLVR) lowers final validation loss for both DeltaNet and Gated DeltaNet.

Problem

• linear attention
• long-context modeling
• self-attention complexity
• recurrent memory
• cross-layer routing

Innovation

• Linear Attention
• Recurrent Memory
• Cross-Layer Routing
• DeltaNet
• Training Throughput