MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs

2026-06-15

AI Summary

This work addresses the significant performance degradation of multimodal large language models with mixture-of-experts (MoE-MLLMs) under mixed-precision quantization, which stems from cross-modal misalignment and intra-visual frequency biases. To mitigate this, the authors propose a modality-decomposed mixed-precision quantization framework that decouples modalities to guide expert selection, filters redundant visual tokens, and employs a modality sensitivity metric to inform bit-width allocation. This approach uniquely integrates modality decomposition with denoised visual frequency statistics to correct biases in expert importance estimation. Experimental results demonstrate that the method incurs an average performance drop of less than 2.9% under W3A16 quantization and achieves particularly notable gains under extreme 2-bit compression.

Abstract

Mixture-of-Experts Multimodal Large Language Models (MoE-MLLMs) offer remarkable performance but incur prohibitive GPU memory costs, making compression essential. Among PTQ methods, expert-level mixed-precision quantization has proven effective for MoE-LLMs, yet suffers notable degradation on MoE-MLLMs due to two overlooked biases in expert importance estimation. (1) At the cross-modal level, the numerical dominance of vision tokens causes expert selection frequency to be dominated by vision tokens, masking experts that are critical to the text modality; (2) at the intra-vision level, the large proportion of redundant vision tokens further skew frequency statistics, obscuring experts critical for informative visual content. To bridge gaps, we propose MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE-MLLMs that decomposes expert selection frequency by modality, filters redundant vision tokens to obtain denoised visual frequency, and further evaluates quantization sensitivity per modality as a complementary signal to frequency-based estimation. These signals are integrated into an Integer Linear Programming formulation to assign per-expert bit-widths under a given budget. Extensive experiments show that MODE is particularly well-suited for MoE-MLLMs, limiting average performance loss to within 2.9% at W3A16, with larger gains at the extreme 2-bit setting.

Problem

• MoE-MLLMs
• mixed-precision quantization
• expert importance estimation
• cross-modal bias
• redundant vision tokens

Innovation

• modality-decomposed quantization
• expert-level mixed-precision
• MoE-MLLMs
• token redundancy filtering
• quantization sensitivity