The Key to Going Linear: Analysis-Driven Transformer Linearization

2026-07-08

AI Summary

This work addresses the severe quadratic complexity of causal self-attention, which hinders the inference efficiency of long-context Transformers, while existing linearization methods struggle to balance efficiency and performance. Under a frozen backbone setting, the study reveals that Softmax attention relies on key-dependent rank-1 orthogonal projections and elucidates why delta-style updates outperform pure gating-based accumulation. Building on these insights, the authors propose a structured intervention strategy—comprising sink tokens, short convolutions, and fixed-budget cache routing—to effectively reduce linear approximation errors. Experiments on LLaMA and Qwen model families (up to 32B parameters) demonstrate that the method surpasses current post-hoc linearization approaches on MMLU and matches the performance of sophisticated adaptive caching frameworks in long-context retrieval tasks.

Abstract

The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in a strict frozen-backbone regime. We show that softmax relies on key-dependent, rank-1 orthogonal projections, elucidating why delta-style networks outperform purely gated accumulation. We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing, which reduces the remaining gap. We scale this linearization approach across LLaMA and Qwen models up to 32B parameters, outperforming prior post hoc baselines on MMLU and matching the long-context retrieval of complex adaptive-caching frameworks.

Problem

• causal self-attention
• quadratic cost
• linearization
• long-context inference
• model quality preservation

Innovation

• Transformer linearization
• causal self-attention
• state update design
• structural interventions
• long-context inference