MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs

2026-02-21

AI Summary

This work addresses the challenge of enabling elastic large language models to dynamically switch quantization precision under varying computational resources, a capability hindered by the sensitivity of post-training quantization (PTQ) calibration parameters to accuracy degradation. The authors propose a token-sensitivity-aware mixed-bit quantization framework that identifies outlier migration as a key cause of calibration instability and introduces, for the first time, a token-level sensitivity-driven dynamic bit allocation mechanism. By integrating recursive residual quantization, token-aware routing, and a Mixture-of-Bits architecture, the method supports multi-precision elastic inference without repeated calibration. Evaluated on LLaMA3-8B, it achieves performance comparable to per-bit calibrated PTQ while significantly enhancing deployment flexibility and efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Changing runtime complexity on cloud and edge devices necessitates elastic large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. However, it has been observed that the calibration parameters for quantization are typically linked to specific precisions, which presents challenges during elastic-precision calibration and precision switching at runtime. In this work, we attribute the source of varying calibration parameters to the varying token-level sensitivity caused by a precision-dependent outlier migration phenomenon.Motivated by this observation, we propose \texttt{MoBiQuant}, a novel Mixture-of-Bits quantization framework that adjusts weight precision for elastic LLM inference based on token sensitivity. Specifically, we propose the many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights and the token-aware router to dynamically select the number of residual bit slices. MoBiQuant enables smooth precision switching while improving generalization for the distribution of token outliers. Experimental results demonstrate that MoBiQuant exhibits strong elasticity, enabling it to match the performance of bit-specific calibrated PTQ on LLaMA3-8B without repeated calibration.

Problem

• elastic LLMs
• quantization
• precision switching
• calibration
• token sensitivity

Innovation

• Mixture-of-Bits
• Elastic LLM
• Token-Adaptive Quantization
• Recursive Residual Quantization
• Outlier Migration