FlexiBit: Fully Flexible Precision Bit-parallel Accelerator Architecture for Arbitrary Mixed Precision AI

2024-11-27

arXiv.org

AI Summary

Current hardware supports only fixed-power-of-two floating-point precisions (e.g., FP8/FP16), limiting adaptability to the mixed-precision requirements of large language models. This restricts computational efficiency for sub-FP8 formats (e.g., FP5/FP6) and necessitates costly hardware redesign for each new precision. To address this, we propose a fully flexible bit-parallel architecture—the first to enable efficient computation across arbitrary non-power-of-two floating-point and integer precisions and formats, achieving complete decoupling among precision, data format, and compute units. Our design comprises three key components: a reconfigurable bit-parallel array, a dynamic precision routing network, and a unified floating-point/integer decoder. Evaluated on GPT-3 FP6 inference, our architecture delivers 1.66× and 1.62× higher performance-per-area than NVIDIA Tensor Cores and BitFusion, respectively, and outperforms state-of-the-art bit-serial architectures by 3.9×.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent research has shown that large language models (LLMs) can utilize low-precision floating point (FP) quantization to deliver high efficiency while maintaining original model accuracy. In particular, recent works have shown the effectiveness of non-power-of-two precisions, such as FP6 and FP5, and diverse sensitivity to low-precision arithmetic of LLM layers, which motivates mixed precision arithmetic including non-power-of-two precisions in LLMs. Although low-precision algorithmically leads to low computational overheads, such benefits cannot be fully exploited due to hardware constraints that support a limited set of power-of-two precisions (e.g., FP8, 16, 32, and 64 in NVIDIA H100 Tensor Core). In addition, the hardware compute units are designed to support standard formats (e.g., E4M3 and E5M2 for FP8). Such practices require re-designing the hardware whenever new precision and format emerge, which leads to high hardware replacement costs to exploit the benefits of new precisions and formats. Therefore, in this paper, we propose a new accelerator architecture, FlexiBit, which efficiently supports FP and INT arithmetic in arbitrary precisions and formats. Unlike previous bit-serial designs, which also provide flexibility but at the cost of performance due to its bit-wise temporal processing nature, FlexiBit's architecture enables bit-parallel processing of any precision and format without compute unit underutilization. FlexiBit's new capability to exploit non-power of two precision and format led to 1.66x and 1.62x higher performance per area on GPT-3 in FP6 targeting a cloud-scale accelerator, compared to a Tensor Core-like architecture and a state-of-the-art bit-parallel flexible precision accelerator, BitFusion, respectively. Also, the bit-parallel nature of FlexiBit's architecture led to 3.9x higher performance/area compared to a state-of-the-art bit-serial architecture.

Problem

• Supports arbitrary mixed precision AI arithmetic efficiently
• Overcomes hardware constraints limiting non-power-of-two precisions
• Eliminates compute unit underutilization in flexible precision processing

Innovation

• FlexiBit enables arbitrary precision and format arithmetic
• Bit-parallel processing avoids compute unit underutilization
• Supports non-power-of-two precisions for higher efficiency