LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation

2026-04-21

AI Summary

Deploying large language models in resource-constrained settings is hindered by substantial computational and memory overhead. This work proposes LBLLM, a lightweight binarization framework that achieves extreme low-bit quantization at W(1+1)A4 through a three-stage distillation process: first initializing a high-quality quantized model via post-training quantization (PTQ), then performing layer-wise distillation of binarized weights while keeping activations in full precision, and finally introducing learnable activation scaling factors to enable dynamic 4-bit activation quantization. The method adopts a decoupled design that separates weight and activation quantization to prevent mutual interference, thereby enhancing training stability and accuracy without requiring additional high-precision pathways or rotation matrices. Trained with only 0.016B tokens on a single GPU, LBLLM surpasses existing W2A4 approaches and sets a new state of the art across language modeling, commonsense question answering, and comprehension benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Deploying large language models (LLMs) in resource-constrained environments is hindered by heavy computational and memory requirements. We present LBLLM, a lightweight binarization framework that achieves effective W(1+1)A4 quantization through a novel three-stage quantization strategy. The framework proceeds as follows: (1) initialize a high-quality quantized model via PTQ; (2) quantize binarized weights, group-wise bitmaps, and quantization parameters through layer-wise distillation while keeping activations in full precision; and (3) training learnable activation quantization factors to dynamically quantize activations to 4 bits. This decoupled design mitigates interference between weight and activation quantization, yielding greater training stability and better inference accuracy. LBLLM, trained only using 0.016B tokens with a single GPU, surpasses existing state-of-the-art binarization methods on W2A4 quantization settings across tasks of language modeling, commonsense QA, and language understanding. These results demonstrate that extreme low-bit quantization of LLMs can be both practical and highly effective without introducing any extra high-precision channels or rotational matrices commonly used in recent PTQ-based works, offering a promising path toward efficient LLM deployment in resource-limited situations.

Problem

• Large Language Models
• Model Binarization
• Low-bit Quantization
• Resource-constrained Deployment
• Efficient Inference

Innovation

• binarization
• three-stage distillation
• low-bit quantization
• activation quantization
• LLM compression