Problem
Research questions and friction points this paper is trying to address.
trillion-parameter MoE
High-Bandwidth Flash
DRAM
bandwidth-to-capacity ratio
host bandwidth
Trillion-Parameter MoE in a Box: Decoupling Memory Provisioning with High-Bandwidth Flash
🤖 AI Summary
研究通过分析万亿参数MoE模型，探讨了使用高带宽闪存（HBF）与DRAM配置解决大规模模型内存供应问题的方法。
📝 Abstract
An MoE appliance for trillion-parameter models at low concurrency must host terabytes of weights on one node and serve prefill and decode with fixed resources. Combining operator analysis of two trillion-parameter MoE models, a measured expert routing trace, and agentic serving traces over multiple turns, we explore a design space spanning High-Bandwidth Flash (HBF) and DRAM configurations, bandwidth exposure, and near-data compute. We find that state bandwidth and HBF transport form two largely orthogonal knees and address two provisioning questions. Q1: Once weights move to HBF, what bandwidth-to-capacity ratio does DRAM require? With a 256-GB floor for the state tier, both models meet a $1.10\times$ completion time target at ratios of only $1.4$--$4.0~\mathrm{s}^{-1}$, roughly an order of magnitude below HBM3e's $33.3~\mathrm{s}^{-1}$. Q2: As HBF internal bandwidth scales with capacity, must bandwidth to the host scale proportionally? Six HBF packages expose 384~GB/s per package to the host, 2.30~TB/s aggregate and 62.5\% below the 6.14~TB/s full exposure reference, while more packages reduce required bandwidth per package at the same target.
Innovation
Methods, ideas, or system contributions that make the work stand out.
High-Bandwidth Flash
Trillion-Parameter MoE
Memory Provisioning
🔎 Similar Papers
No similar papers found.
