Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts

2026-01-29

AI Summary

Existing hybrid Transformer models face limited applicability due to high pretraining costs, substantial data requirements for distillation, and constrained performance on long-context tasks. To address these challenges, this work proposes HypeNet—a novel hybrid architecture that integrates recurrent neural networks with attention mechanisms—augmented by a new positional encoding scheme, HyPE, and an efficient distillation pipeline, HALO. Using only 2.3 billion tokens (approximately 0.01% of the original pretraining data), HypeNet successfully transfers knowledge to the Qwen3 model series. This approach drastically reduces data dependency while preserving overall performance and significantly enhancing both efficiency and generalization on extremely long-context tasks.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Hybrid Transformer architectures, which combine softmax attention blocks and recurrent neural networks (RNNs), have shown a desirable performance-throughput tradeoff for long-context modeling, but their adoption and studies are hindered by the prohibitive cost of large-scale pre-training from scratch. Some recent studies have shown that pre-trained softmax attention blocks can be converted into RNN blocks through parameter transfer and knowledge distillation. However, these transfer methods require substantial amounts of training data (more than 10B tokens), and the resulting hybrid models also exhibit poor long-context performance, which is the scenario where hybrid models enjoy significant inference speedups over Transformer-based models. In this paper, we present HALO (Hybrid Attention via Layer Optimization), a pipeline for distilling Transformer models into RNN-attention hybrid models. We then present HypeNet, a hybrid architecture with superior length generalization enabled by a novel position encoding scheme (named HyPE) and various architectural modifications. We convert the Qwen3 series into HypeNet using HALO, achieving performance comparable to the original Transformer models while enjoying superior long-context performance and efficiency. The conversion requires just 2.3B tokens, less than 0.01% of their pre-training data

Problem

• hybrid attention
• long-context modeling
• knowledge distillation
• Transformer conversion
• efficient architecture

Innovation

• Hybrid Attention
• Knowledge Distillation
• Long-context Modeling
• Positional Encoding
• Efficient Transformers