MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving

2025-02-04

AI Summary

Existing MLLM services employ context caching that only reuses fixed-prefix KV states, rendering them ill-suited for text-image interleaved inputs and dynamic prefix variations in multimodal RAG—leading to substantial redundant recomputation. This work proposes a **position-agnostic multimodal KV caching mechanism**, the first to decouple positional encoding from cached KV states, enabling flexible, cross-sequence and cross-modal KV reuse. Our approach integrates on-demand recomputation, multimodal token alignment, dynamic scheduling, and disk-persistent loading. Experiments demonstrate that, while preserving near-lossless accuracy (<0.3% degradation), our method reduces end-to-end latency by up to 54%, significantly outperforming state-of-the-art caching schemes.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The context caching technique is employed to accelerate the Multimodal Large Language Model (MLLM) inference by prevailing serving platforms currently. However, this approach merely reuses the Key-Value (KV) cache of the initial sequence of prompt, resulting in full KV cache recomputation even if the prefix differs slightly. This becomes particularly inefficient in the context of interleaved text and images, as well as multimodal retrieval-augmented generation. This paper proposes position-independent caching as a more effective approach for multimodal information management. We have designed and implemented a caching system, named MPIC, to address both system-level and algorithm-level challenges. MPIC stores the KV cache on local or remote disks when receiving multimodal data, and calculates and loads the KV cache in parallel during inference. To mitigate accuracy degradation, we have incorporated integrated reuse and recompute mechanisms within the system. The experimental results demonstrate that MPIC can achieve up to 54% reduction in response time compared to existing context caching systems, while maintaining negligible or no accuracy loss.

Problem

• Efficient multimodal context caching
• Reducing KV cache recomputation
• Improving MLLM serving performance

Innovation

• Position-independent multimodal caching
• Parallel KV cache computation
• Integrated reuse and recompute mechanisms