Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit

2026-07-13

AI Summary

Deploying large multimodal models that integrate hybrid sliding window attention (Hybrid SWA), sparse mixture-of-experts (MoE), and multimodal encoders in production environments entails significant computational, memory, and engineering challenges. This work presents MiMo-V2.5, the first production-grade serving system supporting this composite architecture. MiMo-V2.5 introduces a suite of full-stack optimizations—including SWA-aware prefix cache trees, inter-layer KVCache prefetching, KVCache affinity routing, GCache distributed caching, RDMA-optimized networking, GPU-accelerated image preprocessing, and shared multimodal caching—to simultaneously achieve load balancing, O(W) storage complexity, and high cache hit rates. These innovations substantially reduce inference overhead and enable efficient online serving of large-scale multimodal models.

Abstract

We present a full-pipeline inference optimization for the MiMo-V2.5 model family, which combines Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders. While Hybrid SWA can ideally reduce both attention compute and KVCache storage significantly compared to Full Attention, realizing these gains in production requires substantial engineering effort. We systematically optimize the KVCache system with layerwise prefetch, SWA-aware prefix cache trees, and specialized placement strategies, achieving strict $O(W)$ SWA storage and high cache hit rates. We further build GCache, a high-performance distributed cache infrastructure with RDMA-optimized networking, and develop a KVCache-affinity router to reduce computation while preserving load balancing. We also optimize for multimodal inputs, including GPU image preprocessing, parallel video decoding, and multimodal cache sharing. Together, these optimizations constitute the first large-scale LLM serving system in production that efficiently covers the Hybrid SWA + MoE + multimodal composite architecture.

Problem

• Hybrid SWA
• KVCache optimization
• Mixture-of-Experts
• multimodal inference
• LLM serving

Innovation

• Hybrid Sliding Window Attention
• Mixture-of-Experts
• KVCache Optimization
• Multimodal LLM Serving
• Distributed Caching