WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization

2026-05-26

AI Summary

This work addresses the challenge of simultaneously achieving high accuracy and efficiency in large language models under ultra-low-bit quantization, where post-training quantization suffers from significant accuracy degradation, quantization-aware training incurs prohibitive computational costs, and existing mixed-precision strategies lack fine-grained modeling. To overcome these limitations, we propose WINDQuant—the first reinforcement learning–based, column-block–level mixed-precision quantization framework that dynamically allocates bit widths and quantization schemes to individual column blocks under a global memory budget. WINDQuant innovatively integrates proximal policy optimization (PPO), activation-aware calibration, lightweight unit fitting, and explicit effective-bit accounting, enabling substantial accuracy recovery in ultra-low-bit settings without retraining. Experiments demonstrate that WINDQuant consistently outperforms state-of-the-art methods on LLaMA-family models, achieving an optimal trade-off between high accuracy and low optimization overhead at extremely low bitwidths.

Abstract

Quantization is an effective approach to reduce the memory footprint and inference cost of large language models (LLMs), yet maintaining performance in the ultra-low-bit regime remains challenging. Existing post-training methods often suffer from severe accuracy degradation, while quantization-aware training requires costly retraining and additional resources. Moreover, most mixed-precision strategies rely on coarse-grained or heuristic sensitivity analysis that overlooks fine-grained variations within weight matrices. We propose WINDQuant, a reinforcement-learning-based allocation controller for ultra-low-bit LLM quantization. Rather than introducing another low-level quantization operator, WINDQuant learns how to assign bit-widths and quantization treatments to fine-grained column chunks under a global storage budget. By operating at the column-chunk level, WINDQuant enables flexible and fine-grained precision assignment within layers under a global target bit-width. The implementation combines PPO with activation-aware calibration, lightweight per-unit quantizer fitting, and explicit effective-bit accounting of the learned mixed-precision plan. Experiments on LLaMA models demonstrate that WINDQuant achieves competitive performance in ultra-low-bit settings while reducing optimization overhead relative to retraining-based approaches, highlighting reinforcement learning as a practical controller for adaptive mixed-precision quantization.

Problem

• LLM quantization
• mixed-precision
• ultra-low-bit
• fine-grained allocation
• accuracy degradation

Innovation

• mixed-precision quantization
• reinforcement learning
• column-chunk allocation
• ultra-low-bit LLM
• quantization-aware controller