EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture

2026-05-22

AI Summary

This work addresses the computational inefficiency, memory bandwidth bottlenecks, and underutilized hardware inherent in autoregressive decoding of large language models. The authors propose EVA, a novel architecture that, for the first time, directly computes dot products between vector-quantized weights and inputs, thereby reformulating the core decoding operation from a general matrix-vector multiplication (GEMV) into a highly parallelizable general matrix-matrix multiplication (GEMM). EVA further introduces a structured intermediate buffer to enable conflict-free memory access and eliminate the need for weight reconstruction. Through synergistic hardware-software co-design, the approach achieves up to 11.17× speedup and 7.17× higher energy efficiency over state-of-the-art architectures while preserving quantization accuracy.

Abstract

Large Language Models (LLMs) have achieved impressive performance across diverse domains but remain inefficient during the autoregressive decoding phase. Unlike the prefill stage, which employs compute-bound GEMM operations, decoding executes a sequence of small GEMV-like computations that are memory-bound and underutilize modern accelerators. Weight-only vector quantization (VQ) has emerged as an effective compression technique that clusters model weights into a shared codebook and replaces the original weight matrix with low-precision indices, enabling 2-bit-level weight compression. While this approach substantially reduces model size and memory bandwidth, it still suffers from two critical inefficiencies: the low utilization of GEMV computation and frequent memory conflicts during codebook lookups. This paper presents EVA, an efficient vector-quantization-based architecture that addresses both computational and memory bottlenecks in LLM decoding. EVA builds on a simple yet effective insight that combines input-codebook computation with conflict-free memory access. Instead of reconstructing quantized weights from indices, EVA directly performs dot products between input vectors and the weight codebook, transforming LLM decoding from GEMV to GEMM computation. It then performs structured lookups from an intermediate output buffer, eliminating memory bank conflicts. We further design a hardware-software co-optimized architecture specialized for LLM decoding while remaining compatible with conventional prefill execution. Evaluations show that EVA achieves up to 11.17$\times$ speedup and 7.17$\times$ higher energy efficiency compared with the SOTA lookup-based architecture, while preserving arithmetic precision after vector quantization. Our code is available at https://github.com/dbw6/Eva.git.

Problem

• LLM decoding
• vector quantization
• memory bottleneck
• compute inefficiency
• codebook lookup

Innovation

• vector quantization
• LLM decoding
• GEMM acceleration
• memory conflict elimination
• hardware-software co-design