SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models

2026-07-02

AI Summary

This work addresses the challenge of deploying large vision-language models on resource-constrained devices, where high memory consumption and latency stem from massive parameter counts and cross-modal computation. Existing binary quantization methods suffer significant performance degradation due to their neglect of varying weight importance across layers and modalities. To overcome this, we propose a weight importance-aware binarization approach that introduces, for the first time in large vision-language models, a cross-modal and cross-layer saliency mechanism. Specifically, spatial saliency maps derived from Hessian matrices of both textual and visual inputs distinguish critical intra- and inter-modal weights. These maps are integrated via a modality-guided fusion strategy to construct a saliency-aware weighting scheme that guides the binarization optimization. At approximately 1-bit compression, our method substantially outperforms existing post-training quantization baselines across multiple downstream tasks, achieving drastic reductions in storage and computational costs while effectively preserving model accuracy.

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding, yet their enormous parameter scale and cross-modal computation incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices. Binarization offers an attractive solution by drastically reducing storage and computational costs. However, existing binarization methods neglect the varying importance of weights across different layers and modalities. This causes parameters irrelevant to downstream tasks to be unnecessarily retained, whereas modality-critical weights may not be adequately optimized, resulting in significant performance degradation. To address these challenges, we develop a novel \underline{S}ignificance-\underline{A}ware \underline{B}inarization for \underline{L}arge \underline{V}ision-\underline{L}anguage \underline{M}odels (SAB-LVLM). Specifically, after constructing Hessian matrices for textual and visual inputs, we propose a spatial significance map to distinguish full-precision weights activated under a single modality from those activated across modalities. We then devise a modality-guided integration strategy to obtain the significance-aware binarization map, which measures weight significance across layers and modalities. Subsequently, this binarization map is incorporated into the binarization objective as an error reweighting term, and binarization fitting is performed through an alternating significance-weighted update scheme. Extensive experiments illustrate the superiority of our SAB-LVLM over existing binary PTQ methods under an approximately 1-bit compression constraint. Our code is accessible at https://github.com/LyuQi127/SAB_LVLM.

Problem

• Large Vision-Language Models
• Binarization
• Modality
• Weight Significance
• Model Compression

Innovation

• Significance-Aware Binarization
• Large Vision-Language Models
• Modality-Guided Integration
• Hessian-based Weight Significance
• Binary Post-Training Quantization