RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale

2025-05-05

AI Summary

To address the high inference overhead and deployment challenges of softmax-based attention in Transformers, this paper proposes an efficient distillation framework that rapidly converts standard Transformer models (7B/32B/72B) into linear-complexity RWKV-style decoders. The method integrates attention mechanism distillation, structural reparameterization, and lightweight fine-tuning, requiring only 350–700M tokens (<0.005% of original pretraining data) for conversion. It marks the first successful realization of a 72B-scale linear-attention decoder, achieving inference quality nearly on par with the original models and attaining state-of-the-art performance among comparable methods on mainstream benchmarks. The total conversion cost remains under $2,000. All distilled models are publicly released under the Apache 2.0 + Qwen license, enabling broad research and practical adoption.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present Rapid Attention Distillation to Linear Attention Decoders at Scale (RADLADS), a protocol for rapidly converting softmax attention transformers into linear attention decoder models, along with two new RWKV-variant architectures, and models converted from popular Qwen2.5 open source models in 7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens, less than 0.005% of the token count used to train the original teacher models. Converting to our 72B linear attention model costs less than $2,000 USD at today's prices, yet quality at inference remains close to the original transformer. These models achieve state-of-the-art downstream performance across a set of standard benchmarks for linear attention models of their size. We release all our models on HuggingFace under the Apache 2.0 license, with the exception of our 72B models which are also governed by the Qwen License Agreement. Models at https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102 Training Code at https://github.com/recursal/RADLADS-paper

Problem

• Convert softmax attention transformers to linear attention decoders efficiently
• Reduce training tokens needed for conversion to under 0.005%
• Maintain inference quality close to original transformer models

Innovation

• Converts softmax transformers to linear attention decoders
• Requires minimal tokens for distillation process
• Achieves state-of-the-art performance cost-effectively