GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation

2026-06-22

AI Summary

This work addresses the memory bandwidth bottleneck of autoregressive decoding for large language models on edge devices, where existing quantization methods fail to differentiate between the compute-intensive prefill and memory-intensive decode phases. The authors propose GRINQH, a weight-only, activation-aware post-training quantization framework that dynamically allocates per-channel weight precision based on input activation magnitudes, unifying quantization and sparsification. GRINQH introduces a multi-precision nested storage layout and custom GPU kernels, enabling flexible average bitwidths during decoding for the first time. Evaluated on Llama3 and Qwen3, it outperforms state-of-the-art fixed- and mixed-precision baselines, significantly advancing the Pareto frontier of inference speed and generation quality, and achieving efficient 2-bit text generation.

Abstract

Autoregressive decoding with LLMs is primarily bottlenecked by GPU memory bandwidth, especially in edge-computing settings. While quantization is essential for mitigating this bottleneck, most existing methods treat inference as a uniform process and fail to account for the asymmetry between the compute-bound prefill stage and the memory-bound decoding stage. We propose GRINQH (GRaded INput-based Quantization Hierarchy), a weight-only post-training quantization framework that accelerates decoding by unifying quantization and sparsification. GRINQH leverages activation magnitudes as a proxy for computational importance to dynamically assign weight channels to different precision levels, enabling flexible average bit widths during decoding. Evaluated on Llama3 and Qwen3 models, GRINQH outperforms state-of-the-art fixed- and mixed-precision baselines at comparable 3- and 4-bit settings, even enabling effective 2-bit generation. We experimentally verify theoretical speedups by leveraging a hierarchical nested memory layout for multi-precision storage in a custom GPU kernel. Ultimately, GRINQH establishes a new state-of-the-art Pareto frontier for LLM generation, enabling a dynamic trade-off between generation quality and inference speed.

Problem

• LLM generation
• memory bandwidth bottleneck
• quantization
• autoregressive decoding
• edge computing

Innovation

• quantization
• LLM generation
• memory bandwidth
• mixed-precision
• activation-aware