The New LLM Bottleneck: A Systems Perspective on Latent Attention and Mixture-of-Experts

2025-07-21

AI Summary

Traditional Transformer accelerators face a dichotomy: multi-head attention (MHA) is memory-bound, while feed-forward layers are compute-bound—motivating specialized attention hardware. This work observes that emerging architectures—multi-head latent attention (MLA) and mixture-of-experts (MoE)—eliminate this single-bottleneck paradigm, necessitating holistic system balance. We show that MLA increases attention arithmetic intensity by over two orders of magnitude, rendering it compute-intensive; concurrently, dynamic batch-aware MoE expert scheduling aligns expert computation density with that of dense feed-forward layers. Experiments demonstrate that modern GPUs, high-bandwidth interconnects, and distributed scheduling suffice to support such architectures—obviating dedicated attention accelerators. Our core contribution is the identification and empirical validation of a new “balanced-systems-over-specialized-acceleration” paradigm, providing both theoretical grounding and practical guidelines for hardware-algorithm co-design in large language models. (149 words)

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Computational workloads composing traditional Transformer models are starkly bifurcated. Multi-Head Attention (MHA) is memory-bound, with low arithmetic intensity, while feedforward layers are compute-bound. This dichotomy has long motivated research into specialized hardware to mitigate the MHA bottleneck. This paper argues that recent architectural shifts, namely Multi-head Latent Attention (MLA) and Mixture-of-Experts (MoE), challenge the premise of specialized attention hardware. We make two key observations. First, the arithmetic intensity of MLA is over two orders of magnitude greater than that of MHA, shifting it close to a compute-bound regime well-suited for modern accelerators like GPUs. Second, by distributing MoE experts across a pool of accelerators, their arithmetic intensity can be tuned through batching to match that of the dense layers, creating a more balanced computational profile. These findings reveal a diminishing need for specialized attention hardware. The central challenge for next-generation Transformers is no longer accelerating a single memory-bound layer. Instead, the focus must shift to designing balanced systems with sufficient compute, memory capacity, memory bandwidth, and high-bandwidth interconnects to manage the diverse demands of large-scale models.

Problem

• Addressing the memory-bound and compute-bound dichotomy in Transformer models
• Challenging the need for specialized attention hardware with MLA and MoE
• Designing balanced systems for diverse demands of large-scale models

Innovation

• Multi-head Latent Attention increases arithmetic intensity
• Mixture-of-Experts balances computational profile via batching
• Modern accelerators replace specialized attention hardware