Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines

2026-04-17

AI Summary

This work addresses the challenge of excessive peak memory consumption in multimodal large language models when processing high-resolution images or long videos, where the sheer number of visual tokens leads to prohibitive KV cache memory usage during inference. To overcome this bottleneck, the authors propose a structure-aware sequence input compression mechanism that dynamically compresses the KV cache during the prefilling phase, departing from conventional approaches that only compress after full input processing. Leveraging the structural regularities and representational redundancies inherent in multimodal models, this method enables online compression under strict memory constraints. It substantially reduces peak memory footprint within a fixed memory budget while incurring only minimal performance degradation, thereby significantly enhancing the efficiency and practicality of multimodal inference.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal large language models (MLLMs) have recently demonstrated strong capabilities in understanding and generating responses from diverse visual inputs, including high-resolution images and long video sequences. As these models scale to richer visual representations, inference increasingly relies on storing large numbers of vision tokens in the key-value (KV) cache, making memory consumption a central bottleneck. Existing methods address this issue by identifying redundancy in vision tokens and compressing the cache, but such compression is typically applied only after all inputs are processed, resulting in high peak memory usage during the prefill stage. In this work, we show that MLLMs exhibit inherent structural regularities and representational redundancy that can be exploited to control memory growth throughout inference. Based on this insight, we propose a sequential input-compression mechanism that enforces a fixed memory budget by performing structure-aware key-value cache compression during the prefill process. This approach substantially reduces peak memory usage while maintaining generative performance with only minimal degradation, enabling more practical and memory-efficient multimodal inference.

Problem

• multimodal large language models
• peak memory usage
• key-value cache
• vision tokens
• memory bottleneck

Innovation

• multimodal large language models
• key-value cache compression
• peak memory reduction
• structure-aware compression
• sequential input compression