Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression

2025-07-28

AI Summary

To address the challenge of deploying large multimodal models (LMMs) on edge devices, this paper proposes a joint optimization framework integrating adaptive sparsification and KV cache compression. Our method is the first to co-design structured pruning with KV cache quantization, introducing a fine-tuning-free, adaptive search algorithm based on the Tree-structured Parzen Estimator (TPE) to dynamically allocate layer-wise pruning ratios and KV quantization bit-widths, thereby enabling automatic cache resource optimization. Evaluated on LLaVA-1.5 (7B/13B), our approach achieves significantly faster inference speed and substantially reduced memory footprint compared to state-of-the-art methods—including SparseGPT and Wanda—across multiple compression ratios, while incurring negligible accuracy degradation. The core contributions are: (i) a unified modeling of pruning and KV quantization; (ii) a fine-tuning-free, TPE-based adaptive search mechanism; and (iii) an end-to-end efficient compression paradigm tailored for edge deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large multimodal models (LMMs) have advanced significantly by integrating visual encoders with extensive language models, enabling robust reasoning capabilities. However, compressing LMMs for deployment on edge devices remains a critical challenge. In this work, we propose an adaptive search algorithm that optimizes sparsity and KV cache compression to enhance LMM efficiency. Utilizing the Tree-structured Parzen Estimator, our method dynamically adjusts pruning ratios and KV cache quantization bandwidth across different LMM layers, using model performance as the optimization objective. This approach uniquely combines pruning with key-value cache quantization and incorporates a fast pruning technique that eliminates the need for additional fine-tuning or weight adjustments, achieving efficient compression without compromising accuracy. Comprehensive evaluations on benchmark datasets, including LLaVA-1.5 7B and 13B, demonstrate our method superiority over state-of-the-art techniques such as SparseGPT and Wanda across various compression levels. Notably, our framework automatic allocation of KV cache compression resources sets a new standard in LMM optimization, delivering memory efficiency without sacrificing much performance.

Problem

• Compressing large multimodal models for edge deployment
• Optimizing sparsity and KV cache compression adaptively
• Maintaining model accuracy while enhancing efficiency

Innovation

• Adaptive sparsity optimization via search algorithm
• Dynamic KV cache quantization bandwidth adjustment
• Fast pruning without fine-tuning or weight adjustments