Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

2026-06-15

AI Summary

This work addresses the fragile initialization issue in hybrid linear attention models distilled from pretrained Transformers, which often results in poor initial student performance and heavy reliance on large-scale distillation data. To overcome this, the authors propose Taylor-Calibrate, a lightweight initialization method that, for the first time, leverages Taylor expansion–guided attention statistics to systematically initialize the value projection, memory timescale, and gating parameters of Gated DeltaNet. This approach is further enhanced by layer-wise output alignment optimization. The proposed method dramatically improves zero-shot performance—by up to 88×—and reduces the number of training tokens required to recover performance by 4.9–9.2× across diverse teacher architectures and layer-retention strategies.

Abstract

Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models. A practical way to obtain such models is to convert a pretrained Transformer instead of pretraining a new architecture from scratch, but this conversion is still brittle. Simply copying the teacher attention projections into a Gated DeltaNet (GDN) student does not specify the new recurrent decay, write, and output-gating dynamics. As a result, the converted model often starts in a poor dynamical regime and must spend many distillation tokens repairing initialization rather than learning the remaining teacher behavior. We propose Taylor-Calibrate, a lightweight initialization method for hybrid GDN students. The method uses Taylor-guided teacher attention statistics to set the value projection, memory timescale, write gates, and output gate, then applies a short per-layer alignment step to match each converted layer to the teacher output. Across four teacher settings and three retained-layer policies, Taylor-Calibrate gives substantially stronger zero-shot students, with up to an 88x improvement in a representative ablation, and reaches matched recovery targets with 4.9x--9.2x fewer training tokens than naive conversion.

Problem

• hybrid linear attention
• model conversion
• initialization
• attention distillation
• Transformer

Innovation

• Taylor-Calibrate
• hybrid linear attention
• model distillation
• principled initialization
• Gated DeltaNet