LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

2026-07-30

AI Summary

This work addresses the accuracy degradation and energy-efficiency bottlenecks in low-bit quantized inference of large language models caused by rotation operations. To this end, the authors propose LightRot, a lightweight rotation scheme that innovatively integrates Grouped Local Rotation with Outlier Direction Aligning, alongside a dedicated hardware accelerator based on hierarchical Fast Hadamard Transform (FHT). This approach achieves the first co-optimization of a lightweight rotation mechanism and 28nm CMOS hardware, enabling efficient and high-accuracy execution of dialogue tasks for models such as LLaMA2/3 at 4-bit precision. Experimental results demonstrate strong real-world applicability on MT-Bench, with a peak energy efficiency of 27.4 TOPS/W, significantly outperforming existing solutions.

Abstract

As large language models (LLMs) continue to demonstrate exceptional capabilities across various domains, the challenge of achieving energy-efficient and accurate inference becomes increasingly critical. This work presents LightRot, a lightweight rotation scheme and dedicated hardware accelerator designed for low-bit LLM inference. The proposed architecture integrates Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) algorithms with a hierarchical Fast Hadamard Transform (FHT)-based rotation unit to address key challenges in low-bit quantization, including the energy overhead of rotation operations. The proposed accelerator, implemented in a 28nm CMOS process, achieves a peak energy efficiency of 27.4 TOPS/W for 4-bit inference, surpassing prior state-of-the-art designs. Unlike conventional approaches that rely on higher-precision inference or evaluate on basic language modeling tasks like GPT-2, LightRot is optimized for advanced models such as LLaMA2-13B and LLaMA3-8B. Its performance is further validated on MT-Bench, demonstrating robust applicability to real-world conversational scenarios and redefining benchmarks for chat-based AI systems. By synergizing algorithmic innovations and hardware efficiency, this work sets a new paradigm for scalable, low-bit LLM inference, paving the way for sustainable AI advancements.

Problem

• low-bit inference
• large language models
• energy efficiency
• quantization
• hardware acceleration

Innovation

• LightRot
• low-bit quantization
• rotation scheme
• hardware accelerator
• energy efficiency