BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models

2026-02-04

AI Summary

This work addresses the challenge of deploying large language models on resource-constrained devices, where high memory consumption and bandwidth requirements pose significant barriers, and existing post-training quantization methods suffer severe performance degradation at 2–3 bits. The authors propose Bit-Plane Decomposition Quantization (BPDQ), a novel approach that constructs adaptive quantization grids using bit planes and scalar coefficients, iteratively optimized with approximate second-order information to align the quantization process with output error minimization in a Hessian-induced geometric space. This strategy effectively expands the feasible solution space and enables efficient error compensation. Experiments demonstrate that BPDQ successfully deploys Qwen2.5-72B on a single RTX 3090 GPU at 2 bits, achieving 83.85% accuracy on GSM8K—compared to 90.83% with 16-bit precision.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language model (LLM) inference is often bounded by memory footprint and memory bandwidth in resource-constrained deployments, making quantization a fundamental technique for efficient serving. While post-training quantization (PTQ) maintains high fidelity at 4-bit, it deteriorates at 2-3 bits. Fundamentally, existing methods enforce a shape-invariant quantization grid (e.g., the fixed uniform intervals of UINT2) for each group, severely restricting the feasible set for error minimization. To address this, we propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients, and iteratively refines them using approximate second-order information while progressively compensating quantization errors to minimize output discrepancy. In the 2-bit regime, BPDQ enables serving Qwen2.5-72B on a single RTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit). Moreover, we provide theoretical analysis showing that the variable grid expands the feasible set, and that the quantization process consistently aligns with the optimization objective in Hessian-induced geometry. Code: github.com/KingdalfGoodman/BPDQ.

Problem

• large language models
• post-training quantization
• low-bit quantization
• memory-constrained inference
• quantization grid

Innovation

• Bit-Plane Decomposition
• Variable Quantization Grid
• Post-Training Quantization
• Second-Order Optimization
• Low-Bit LLM Inference