🤖 AI Summary
This work addresses the memory capacity and bandwidth limitations of KV caching in large language model inference, which hinder simultaneous support for terabyte-scale memory and hundreds of gigabytes per second throughput. The authors propose a novel optoelectronic hybrid CXL memory architecture that introduces passive optical interconnects into a CXL memory pool, enabling a switchless full-mesh topology that allows 16 hosts to share 32 TB of memory. By eliminating electrical switches, the design circumvents bottlenecks in latency, reach, and power consumption, yielding a scalable low-latency shared memory system. Experimental results demonstrate over 50% lower system latency compared to an all-electrical CXL baseline and a 6.6× speedup in first-token generation time under multi-turn dialogue scenarios, effectively mitigating the performance cliff caused by cache eviction.
📝 Abstract
LLM inference at scale faces a memory wall. The KV cache demands tens of terabytes at hundreds of gigabytes per second, yet no current memory tier delivers both at once. Characterization across multi-generation GPU systems with various LLaMA models shows host memory retrieval achieves up to 100x speedup over re-computation but supports only tens of concurrent long-context users. Electrical CXL pooling theoretically bridges this gap, but switch latency, cable reach limits, and power-scaling issues prevent practical TB-scale deployments. We present the Marvell Photonic Fabric Memory Appliance, a photonic-CXL hybrid architecture replacing electrical switches with a passive fiber shuffle to deliver 32 TB shared memory across 16 hosts via a switch-free full- crossbar topology. Emulation results demonstrate over 50 percent latency reduction versus electrical CXL pools. Simulation results show that the PF Memory Appliance eliminates cache eviction cliffs by improving time-to-first-token by 6.6x for multi-turn conversations workloads.