Qwen2.5-1M Technical Report

2025-01-26

AI Summary

Large language models (LLMs) suffer from weak long-context modeling, high inference costs, and degraded short-context performance when extended to longer sequences. Method: We propose an efficient LLM series supporting up to 1 million tokens of context. Our approach introduces the first training-free context-length extrapolation technique—enabling >4× context extension—integrates sparse attention with block-wise prefill optimization, and designs a multi-level inference engine featuring GPU kernel optimization, pipelined parallelism, and dynamic scheduling. Long-context capability is enhanced via synthetic long-sequence data generation, progressive pretraining, and multi-stage supervised fine-tuning. Results: Qwen2.5-14B-Instruct-1M significantly outperforms GPT-4o-mini on long-context benchmarks, supporting 8× longer contexts. In million-token scenarios, prefill throughput improves by 3–7×, while short-context accuracy remains uncompromised. The code and models are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

We introduce Qwen2.5-1M, a series of models that extend the context length to 1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series have significantly enhanced long-context capabilities through long-context pre-training and post-training. Key techniques such as long data synthesis, progressive pre-training, and multi-stage supervised fine-tuning are employed to effectively enhance long-context performance while reducing training costs. To promote the use of long-context models among a broader user base, we present and open-source our inference framework. This framework includes a length extrapolation method that can expand the model context lengths by at least four times, or even more, without additional training. To reduce inference costs, we implement a sparse attention method along with chunked prefill optimization for deployment scenarios and a sparsity refinement method to improve precision. Additionally, we detail our optimizations in the inference engine, including kernel optimization, pipeline parallelism, and scheduling optimization, which significantly enhance overall inference performance. By leveraging our inference framework, the Qwen2.5-1M models achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million tokens of context. This framework provides an efficient and powerful solution for developing applications that require long-context processing using open-source models. The Qwen2.5-1M series currently includes the open-source models Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly improved in long-context tasks without compromising performance in short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model significantly outperforms GPT-4o-mini in long-context tasks and supports contexts eight times longer.

Problem

• Long Sentence Processing
• Language Model Performance
• Cost and Speed Optimization

Innovation

• Long-sequence Processing
• Sparse Attention
• Multi-stage Fine-tuning