MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration

2026-06-16

AI Summary

This work addresses the hardware inefficiency in large language model inference caused by nonlinear normalization operations—such as LayerNorm, RMSNorm, and Softmax—which typically rely on dedicated hardware modules, leading to resource redundancy and excessive silicon area consumption. To overcome this limitation, the authors propose MIVE (Minimalist Integer Vector Engine), a unified programmable architecture that integrates all three normalization functions into a single design. By leveraging a shared data path, integer arithmetic, and reusable computation patterns, MIVE enables extensive hardware resource sharing across these operations. ASIC implementation results demonstrate that MIVE achieves significantly improved area efficiency and energy efficiency while supporting multifunctional normalization, outperforming existing specialized accelerators.

Abstract

The rapid growth of Large Language Models (LLMs) has intensified the need for specialized hardware accelerators that can satisfy stringent inference latency and power constraints. Although matrix multiplications dominate the overall computational workload, non-linear vector normalization operations, such as LayerNorm, RMSNorm and Softmax can become critical hardware bottlenecks. Existing accelerators typically implement these functions using dedicated hardware blocks, leading to duplicated resources and inefficient silicon utilization. To address this limitation, we propose a Minimalist Integer Vector Engine (MIVE), a programmable architecture capable of executing all three operations within a unified datapath. By exploiting common computational patterns across LayerNorm, RMSNorm and Softmax the proposed vector engine maximizes hardware sharing while reducing implementation overhead. Physical ASIC implementation results show that MIVE provides comprehensive multi-function support while achieving higher area and hardware efficiency than most state-of-the-art standalone accelerators.

Problem

• LayerNorm
• RMSNorm
• Softmax
• hardware accelerator
• silicon efficiency

Innovation

• Minimalist Integer Vector Engine
• LayerNorm
• RMSNorm
• Softmax
• hardware sharing