From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs

2026-01-07

arXiv.org

AI Summary

Deploying large language models on resource-constrained hardware faces significant challenges, including the trade-off between accuracy and efficiency and the complexity of quantization hyperparameter tuning. This work proposes the Hardware-Aware Quantization Agent (HAQA), which leverages a large language model to automatically optimize quantization hyperparameters and adapt them to target hardware, enabling cross-platform adaptive quantization strategies. By doing so, HAQA substantially reduces the need for manual intervention and streamlines the deployment pipeline. Experiments on the Llama family of models demonstrate that HAQA achieves up to 2.3× speedup in inference latency and throughput while maintaining or even improving model accuracy, outperforming conventional non-optimized deployment approaches.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Deploying models, especially large language models (LLMs), is becoming increasingly attractive to a broader user base, including those without specialized expertise. However, due to the resource constraints of certain hardware, maintaining high accuracy with larger model while meeting the hardware requirements remains a significant challenge. Model quantization technique helps mitigate memory and compute bottlenecks, yet the added complexities of tuning and deploying quantized models further exacerbates these challenges, making the process unfriendly to most of the users. We introduce the Hardware-Aware Quantization Agent (HAQA), an automated framework that leverages LLMs to streamline the entire quantization and deployment process by enabling efficient hyperparameter tuning and hardware configuration, thereby simultaneously improving deployment quality and ease of use for a broad range of users. Our results demonstrate up to a 2.3x speedup in inference, along with increased throughput and improved accuracy compared to unoptimized models on Llama. Additionally, HAQA is designed to implement adaptive quantization strategies across diverse hardware platforms, as it automatically finds optimal settings even when they appear counterintuitive, thereby reducing extensive manual effort and demonstrating superior adaptability. Code will be released.

Problem

• large language models
• model quantization
• hardware constraints
• deployment
• accuracy

Innovation

• Hardware-Aware Quantization
• LLM-based Agent
• Automated Deployment
• Adaptive Quantization
• Model Compression