SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference

2025-10-20

AI Summary

In Transformer inference, Softmax and LayerNorm impose significant computational and memory overheads, hindering real-time performance. This paper proposes SOLE, a hardware-software co-design framework featuring two novel approximate operators: E2Softmax and AILayerNorm. Without requiring model retraining, SOLE achieves high-accuracy, low-overhead approximation via log₂-based quantization, logarithmic-domain division approximation, and low-precision statistic computation—enabling both low-precision arithmetic and sub-8-bit storage. Evaluated against GPU baselines, SOLE delivers multiple-fold improvements in inference throughput and energy efficiency. Against state-of-the-art custom accelerators, it achieves 3.04×–3.86× higher energy efficiency and 2.82×–3.32× better area efficiency. The core contribution is the first retraining-free, hardware-software co-optimized acceleration solution for Transformer inference that simultaneously balances accuracy, latency, energy, and silicon area.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Transformers have shown remarkable performance in both natural language processing (NLP) and computer vision (CV) tasks. However, their real-time inference speed and efficiency are limited due to the inefficiency in Softmax and Layer Normalization (LayerNorm). Previous works based on function approximation suffer from inefficient implementation as they place emphasis on computation while disregarding memory overhead concerns. Moreover, such methods rely on retraining to compensate for approximation error which can be costly and inconvenient. In this paper, we present SOLE, a hardware-software co-design for Softmax and LayerNorm which is composed of E2Softmax and AILayerNorm. E2Softmax utilizes log2 quantization of exponent function and log-based division to approximate Softmax while AILayerNorm adopts low-precision statistic calculation. Compared with state-of-the-art designs, we achieve both low-precision calculation and low bit-width storage on Softmax and LayerNorm. Experiments show that SOLE maintains inference accuracy without retraining while offering orders of magnitude speedup and energy savings over GPU, achieving 3.04x, 3.86x energy-efficiency improvements and 2.82x, 3.32x area-efficiency improvements over prior state-of-the-art custom hardware for Softmax and LayerNorm, respectively.

Problem

• Optimizes inefficient Softmax and LayerNorm in Transformer inference
• Reduces memory overhead and computation costs through hardware-software co-design
• Maintains accuracy without retraining while improving speed and energy efficiency

Innovation

• Hardware-software co-design for Softmax and LayerNorm
• Log2 quantization and log-based division for Softmax
• Low-precision statistic calculation for LayerNorm