Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models

2026-05-19

AI Summary

This work addresses the significant performance degradation of large vision-language models under low-bit quantization, which stems from the divergent activation distributions between visual and textual modalities. To mitigate this issue, the authors propose SplitQ, a novel post-training quantization framework featuring a Modality-specific Outlier Channel Decoupling (MOCD) module that separates heterogeneous modality activations via channel splitting, thereby isolating anomalous activations. SplitQ further incorporates an Adaptive Cross-modal Calibration (ACC) mechanism to dynamically compensate for quantization errors. Requiring only lightweight, dual-branch learnable calibration, the method achieves remarkable efficiency and accuracy: under extremely low-bit settings such as W3A3, it outperforms existing approaches and attains 93.5% of FP16 performance—69.5 versus 74.3—across six mainstream multimodal benchmarks.

Abstract

Low-bit post-training quantization (PTQ) is a pivotal technique for deploying Vision-Language Models (VLMs) on resource-constrained devices. However, existing PTQ methods often degrade VLMs' accuracy due to the heterogeneous activation distributions of text and vision modalities during quantization. We find that this cross-modal heterogeneity is distributed unevenly across channels: a small subset of channels contains most modality-specific outliers, and these outliers typically reside in different channels for each modality. Motivated by this, we propose SplitQ, a channel-Splitting-driven post-training Quantization framework. At its core, SplitQ introduces a novel Modality-specific Outlier Channel Decoupling (MOCD) module that effectively isolates salient modality-specific outlier channels with minimal overhead. To further address the remaining cross-modal distribution discrepancies, we design an Adaptive Cross-Modal Calibration (ACC) module that employs dual lightweight learnable branches to dynamically mitigate modality-induced quantization errors. Extensive experiments on popular VLMs demonstrate that SplitQ significantly outperforms existing approaches across 6 popular multi-modal datasets under all evaluated quantization settings, including W4A8, W4A4, W3A3, and W3A2. Notably, SplitQ preserves 93.5% of FP16 performance under the challenging W3A3 setting (69.5 vs. 74.3), pushing the efficiency frontier for deploying advanced VLMs. Our code is available at https://github.com/EMVision-NK/SplitQ

Problem

• modality heterogeneity
• low-bit quantization
• vision-language models
• post-training quantization
• cross-modal distribution

Innovation

• post-training quantization
• vision-language models
• modality heterogeneity
• outlier channel decoupling
• cross-modal calibration