Aquifer: Hierarchical Memory Pooling with CXL and RDMA for MicroVM Snapshots

2026-06-22

AI Summary

This work addresses the dual challenges of 25–35% DRAM waste due to memory fragmentation in cloud environments and high cold-start latency during MicroVM snapshot restoration. To tackle these issues, the authors propose a hierarchical memory pooling architecture that synergistically integrates CXL and RDMA. Snapshot pages are categorized by access热度 (hotness): hot pages reside in a low-latency CXL-attached pool, while cold and zero pages are stored in an RDMA-based remote memory pool. The design includes a novel ownership coherence protocol tailored for non-cache-coherent CXL devices to ensure correctness in multi-host sharing scenarios. Key innovations include the first coordinated use of CXL and RDMA for MicroVM snapshot serving, a hotness-aware snapshot format, and a hybrid loading mechanism featuring hot-page preloading with asynchronous on-demand fetching of cold pages. Experiments demonstrate that the end-to-end restoration latency is reduced by 2.2× on average compared to Firecracker and outperforms the state-of-the-art by 1.1×.

Abstract

Memory stranding wastes 25-35% of installed DRAM in production cloud clusters. Memory pooling over CXL and RDMA offers a remedy, but neither technology alone suffices: CXL provides low-latency, load/store-transparent access limited to a pod, while RDMA provides cluster-wide reach at higher latency with software overhead. A hierarchical architecture combining both tiers is the practical path forward, yet remains unexplored for MicroVM-based serverless computing, where snapshot restore latency is the dominant cold-start bottleneck. We present Aquifer, the first system to serve MicroVM snapshots from a hierarchical CXL+RDMA memory pool. A characterization of snapshot images reveals that the vast majority of pages are either zero or cold, enabling a hotness-based snapshot format that eliminates zero pages and places only the hot working set in the CXL pool while storing cold pages in the RDMA pool. Sharing these snapshots across hosts on CXL 2.0 multi-headed devices, which lack hardware cache coherence, requires Aquifer's ownership-based coherence protocol to ensure correctness. Finally, Aquifer uses a copy-based page serving mechanism pre-installs hot pages from CXL memory before MicroVM resume and demand-pages cold pages asynchronously from RDMA. On emulated CXL+RDMA hardware, Aquifer achieves a 2.2x geometric-mean speedup in end-to-end invocation time over Firecracker and 1.1x over the next best alternative.

Problem

• Memory pooling
• MicroVM snapshots
• Cold-start latency
• CXL
• RDMA

Innovation

• hierarchical memory pooling
• CXL
• RDMA
• MicroVM snapshots
• ownership-based coherence