Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models

2025-03-20

AI Summary

Video large language models (VideoLLMs) suffer from explosive KV cache memory consumption when processing long videos, severely hindering inference efficiency. To address this, we propose VidKV—the first plug-and-play, ultra-low-bit (1–1.58-bit) KV cache quantization method specifically designed for VideoLLMs. Our approach introduces hybrid-precision, channel-wise quantization for key caches: outlier channels are quantized to 2 bits, while regular channels employ 1-bit quantization augmented with FFT-based compression; value caches are quantized to 1.58 bits per channel and further enhanced by semantic-aware selection of critical visual tokens. Evaluated on LLaVA-OV-7B and Qwen2.5-VL-7B across six benchmark datasets, VidKV achieves average KV cache bitrates of 1.5 bits (keys) and 1.58 bits (values), with near-lossless performance relative to FP16—while substantially reducing GPU memory footprint.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Video large language models (VideoLLMs) have demonstrated the capability to process longer video inputs and enable complex reasoning and analysis. However, due to the thousands of visual tokens from the video frames, key-value (KV) cache can significantly increase memory requirements, becoming a bottleneck for inference speed and memory usage. KV cache quantization is a widely used approach to address this problem. In this paper, we find that 2-bit KV quantization of VideoLLMs can hardly hurt the model performance, while the limit of KV cache quantization in even lower bits has not been investigated. To bridge this gap, we introduce VidKV, a plug-and-play KV cache quantization method to compress the KV cache to lower than 2 bits. Specifically, (1) for key, we propose a mixed-precision quantization strategy in the channel dimension, where we perform 2-bit quantization for anomalous channels and 1-bit quantization combined with FFT for normal channels; (2) for value, we implement 1.58-bit quantization while selectively filtering semantically salient visual tokens for targeted preservation, for a better trade-off between precision and model performance. Importantly, our findings suggest that the value cache of VideoLLMs should be quantized in a per-channel fashion instead of the per-token fashion proposed by prior KV cache quantization works for LLMs. Empirically, extensive results with LLaVA-OV-7B and Qwen2.5-VL-7B on six benchmarks show that VidKV effectively compresses the KV cache to 1.5-bit and 1.58-bit precision with almost no performance drop compared to the FP16 counterparts.

Problem

• Reduces memory usage in VideoLLMs via KV cache quantization.
• Explores sub-2-bit KV cache quantization for VideoLLMs.
• Proposes VidKV for efficient KV cache compression without performance loss.

Innovation

• Mixed-precision quantization for key channels
• 1.58-bit quantization for value channels
• Per-channel quantization for VideoLLMs value cache