BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression

2026-07-09

AI Summary

This work addresses the challenge of compressing large language models to extremely low bitwidths (e.g., 2 bits per weight), where memory, bandwidth, and storage constraints often compromise both representational capacity and efficiency. The authors propose Binary Spherical Quantization (BSQ), a codebook-free framework that maps local weight blocks onto the unit hypersphere and directly binarizes them into compact spherical codes. By integrating residual rate-distortion optimization, class-aware recovery distillation, an 8-bit protected channel, and LoRA adapter fusion, BSQ achieves highly efficient compression within a fixed storage budget. Notably, this approach is the first to eliminate explicit codebooks and index lookups, substantially reducing model size while preserving strong inference performance even at ultra-low bitwidths.

Abstract

Large language models (LLMs) are increasingly constrained by memory capacity, weight bandwidth, and checkpoint storage during deployment. Existing low-bit compression methods mainly follow two directions. Scalar or group-wise quantization is simple and compatible with efficient low-precision kernels, but its representation capacity becomes limited when the target budget approaches 2 bits per weight. Vector-quantized weight compression provides a richer block-level representation, but usually introduces explicit codebooks, index lookup, and additional storage accounting. This paper presents BiSCo-LLM, a codebook-free binary spherical coding framework for extreme low-bit LLM weight compression. The core pipeline is built on three components. First, local weight chunks are mapped onto a unit hypersphere and binarized into compact spherical codes, so that the main payload is a bit-packed sign stream rather than explicit VQ centroids. Second, a residual BSQ stage encodes the reconstruction error left by the base spherical codec, providing an explicit rate-distortion path without stored codebooks. Third, category-wise recovery distillation is performed after replacing each Transformer module category, reducing the mismatch between local weight reconstruction and assembled model behavior. A small 8-bit protected-channel path is used as an auxiliary stabilization mechanism for sensitive channels and is counted separately from the BSQ payload. The reported storage budget includes binary codes, neural decoders, protected-channel payloads, LoRA adapters, and metadata.

Problem

• Large Language Model Compression
• Extreme Low-Bit Quantization
• Vector Quantization
• Codebook-Free Compression
• Memory Efficiency

Innovation

• Binary Spherical Coding
• Codebook-Free Compression
• Extreme Low-Bit Quantization
• Residual BSQ
• Recovery Distillation