MiMo-V2-Flash Technical Report

2026-01-06

arXiv.org

AI Summary

This work proposes a 309B-parameter sparse mixture-of-experts (MoE) language model with only 15B activated parameters per token, designed to enhance reasoning speed, capability, and agent-task performance while reducing computational costs. The architecture integrates sliding-window and global attention mechanisms and introduces a multi-token prediction (MTP) framework alongside a multi-teacher online policy distillation (MOPD) approach to enable efficient training and speculative decoding. Despite using merely one-half to one-third of the activated parameters compared to leading open-source models of similar scale, the proposed model achieves comparable or superior performance, accelerates inference by up to 2.6×, and supports context lengths of up to 3.6 million tokens.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present MiMo-V2-Flash, a Mixture-of-Experts (MoE) model with 309B total parameters and 15B active parameters, designed for fast, strong reasoning and agentic capabilities. MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention, with a 128-token sliding window under a 5:1 hybrid ratio. The model is pre-trained on 27 trillion tokens with Multi-Token Prediction (MTP), employing a native 32k context length and subsequently extended to 256k. To efficiently scale post-training compute, MiMo-V2-Flash introduces a novel Multi-Teacher On-Policy Distillation (MOPD) paradigm. In this framework, domain-specialized teachers (e.g., trained via large-scale reinforcement learning) provide dense and token-level reward, enabling the student model to perfectly master teacher expertise. MiMo-V2-Flash rivals top-tier open-weight models such as DeepSeek-V3.2 and Kimi-K2, despite using only 1/2 and 1/3 of their total parameters, respectively. During inference, by repurposing MTP as a draft model for speculative decoding, MiMo-V2-Flash achieves up to 3.6 acceptance length and 2.6x decoding speedup with three MTP layers. We open-source both the model weights and the three-layer MTP weights to foster open research and community collaboration.

Problem

• Mixture-of-Experts
• fast reasoning
• agentic capabilities
• large language model
• efficient inference

Innovation

• Mixture-of-Experts
• Hybrid Attention
• Multi-Token Prediction
• On-Policy Distillation
• Speculative Decoding