LLM Compression: How Far Can We Go in Balancing Size and Performance?

2025-08-15

AI Summary

This study investigates the feasibility and performance limits of 4-bit quantization for lightweight deployment of large language models (LLMs). Addressing information retrieval, Boolean question answering, and mathematical reasoning—three representative tasks—we systematically compare Group Scaling Quantization (GSQ) and GPTQ, two state-of-the-art 4-bit quantization methods, across LLaMA, Qwen, and Phi model families. Evaluations are conducted on MS MARCO, BoolQ, and GSM8K benchmarks. Our experiments reveal, for the first time, systematic trade-offs between quantization accuracy, model scale, and task type: under 4-bit quantization, models retain over 90% of original accuracy while achieving ~75% memory reduction, 40–60% lower inference latency, and 1.8–2.3× higher throughput. These findings establish a reproducible quantization benchmark and provide practical guidelines for efficient LLM deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Quantization is an essential and popular technique for improving the accessibility of large language models (LLMs) by reducing memory usage and computational costs while maintaining performance. In this study, we apply 4-bit Group Scaling Quantization (GSQ) and Generative Pretrained Transformer Quantization (GPTQ) to LLaMA 1B, Qwen 0.5B, and PHI 1.5B, evaluating their impact across multiple NLP tasks. We benchmark these models on MS MARCO (Information Retrieval), BoolQ (Boolean Question Answering), and GSM8K (Mathematical Reasoning) datasets, assessing both accuracy and efficiency across various tasks. The study measures the trade-offs between model compression and task performance, analyzing key evaluation metrics, namely accuracy, inference latency, and throughput (total output tokens generated per second), providing insights into the suitability of low-bit quantization for real-world deployment. Using the results, users can then make suitable decisions based on the specifications that need to be met. We discuss the pros and cons of GSQ and GPTQ techniques on models of different sizes, which also serve as a benchmark for future experiments.

Problem

• Evaluate 4-bit quantization impact on LLM performance
• Assess trade-offs between model compression and task accuracy
• Compare GSQ and GPTQ techniques for real-world deployment

Innovation

• 4-bit Group Scaling Quantization for LLMs
• Generative Pretrained Transformer Quantization technique
• Evaluating accuracy and efficiency trade-offs