AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving

2026-04-28

AI Summary

Traditional LLM inference systems are GPU-centric and struggle to efficiently handle memory-bound attention computations in long-context decoding, resulting in high latency and poor energy efficiency. This work proposes AMMA—a memory-centric, multi-chiplet architecture that shifts the system’s focus from the GPU to an HBM-PNM cube. By integrating a lightweight logic-core microarchitecture, a two-level hybrid parallelism strategy, and a reordered collective communication mechanism, AMMA achieves substantial performance gains with minimal power and area overhead. Compared to the NVIDIA H100, AMMA reduces attention latency by 15.5× and energy consumption by 6.9×.

Abstract

All current LLM serving systems place the GPU at the center, from production-level attention-FFN disaggregation to NVIDIA's Rubin GPU-LPU heterogeneous platform. Even academic PIM/PNM proposals still treat the GPU as the central hub for cross-device communication. Yet the GPU's compute-rich architecture is fundamentally mismatched with the memory-bound nature of decode-phase attention, inflating serving latency while wasting power and die area on idle compute units. The problem is compounded as reasoning and agentic workloads push context lengths toward one million tokens, making attention latency the primary user-facing bottleneck. To address these inefficiencies, we present AMMA, a multi-chiplet, memory-centric architecture for low-latency long-context attention. AMMA replaces GPU compute dies with HBM-PNM cubes, roughly doubling the available memory bandwidth to better serve memory-bound attention workloads. To translate this bandwidth into proportional performance gains, we introduce (i) a logic-die microarchitecture that fully exploits per-cube internal bandwidth for decode attention under a minimal power and area budget, (ii) a two-level hybrid parallelism scheme, and (iii) a reordered collective flow that reduces intra-chip die-to-die communication overhead. We further conduct a design-space exploration over per-cube compute power and intra-chip D2D link bandwidth, providing actionable guidance for hardware designers. Evaluations show that AMMA achieves 15.5X lower attention latency and 6.9X lower energy consumption compared with the NVIDIA H100.

Problem

• LLM serving
• attention latency
• memory-bound
• long-context
• GPU-centric architecture

Innovation

• memory-centric architecture
• HBM-PNM
• multi-chiplet
• long-context attention
• decode-phase optimization