Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs

2025-09-14

AI Summary

Quantization and pruning—two key model compression techniques for large language models (LLMs)—exhibit conflicting distributional preferences: quantization favors compact weight distributions, whereas pruning benefits from high-variance ones. Method: This paper proposes Optimal Brain Restoration (OBR), a unified, training-free framework that jointly optimizes quantization and sparsification. Leveraging Hessian-based second-order information, OBR formulates a principled objective and introduces proxy approximations along with grouped second-order gradient error compensation, enabling derivation of a closed-form solution for synergistic alignment. Contribution/Results: Under W4A4KV4 quantization combined with 50% structured sparsity, OBR achieves up to 4.72× inference speedup and 6.4× memory reduction over FP16 dense models, significantly improving compression ratio and deployment efficiency without retraining.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent advances in Large Language Model (LLM) compression, such as quantization and pruning, have achieved notable success. However, as these techniques gradually approach their respective limits, relying on a single method for further compression has become increasingly challenging. In this work, we explore an alternative solution by combining quantization and sparsity. This joint approach, though promising, introduces new difficulties due to the inherently conflicting requirements on weight distributions: quantization favors compact ranges, while pruning benefits from high variance. To attack this problem, we propose Optimal Brain Restoration (OBR), a general and training-free framework that aligns pruning and quantization by error compensation between both. OBR minimizes performance degradation on downstream tasks by building on a second-order Hessian objective, which is then reformulated into a tractable problem through surrogate approximation and ultimately reaches a closed-form solution via group error compensation. Experiments show that OBR enables aggressive W4A4KV4 quantization with 50% sparsity on existing LLMs, and delivers up to 4.72x speedup and 6.4x memory reduction compared to the FP16-dense baseline.

Problem

• Combining quantization and sparsity for LLM compression
• Resolving conflicting weight distribution requirements
• Minimizing performance degradation through error compensation

Innovation

• Joint quantization and sparsity framework
• Training-free error compensation method
• Second-order Hessian objective optimization