MemExchange: Cloud-Scale Memory Trading

2026-07-13

AI Summary

This work addresses the coexistence of memory over-provisioning and per-tenant memory constraints in cloud data centers, which leads to low aggregate utilization and high cache miss rates. The authors propose a cluster-scale, multi-tenant memory management system that, for the first time, enables decentralized, coordination-free memory pooling across nodes without requiring forced co-location. The system leverages online miss-rate curves (MRCs) to estimate marginal utility and dynamically adjusts cache allocations per tenant. It also introduces an application-layer MTC protocol that supports remote, one-sided RDMA operations, eliminating the need for remote CPU involvement. Experimental results on a hundred-node cluster demonstrate that remote access overhead is reduced by 2.3× compared to TCP, rack-level memory utilization improves by 13%, and cache miss rates for memory-constrained tenants under skewed workloads decrease by up to 63%.

Abstract

To handle unpredictable workloads, cloud providers typically over-provision memory to meet peak demand, resulting in substantial underutilization across datacenter clusters. At the same time, memory-constrained tenants may suffer elevated cache miss rates, even when idle capacity remains stranded elsewhere in the infrastructure. MemExchange is a cluster-wide, multi-tenant memory management system that dynamically right-sizes in-memory caching tenants according to workload demand. Leveraging marginal-utility-based allocation derived from online Miss Ratio Curve (MRC) estimation, MemExchange redistributes idle memory between tenants across physical nodes using RDMA. This approach transforms the dedicated caching memory scattered across servers into a logically aggregated pool, enabling cross-node memory exchange without centralized coordination or forced tenant co-location. To support efficient remote access, we design the MemExchange Tracker Communication (MTC) protocol, an application-layer mechanism that coordinates memory reallocation and enables one-sided RDMA operations without involving remote CPUs. We implement MemExchange in Memcached and evaluate it through microbenchmarks, medium and rack-scale deployments of up to 100 CloudLab servers. Our results show up to 2.3x lower remote-access overhead compared to TCP-based designs, a 13% increase in cluster-wide memory utilization at rack scale, and up to 63% reduction in miss rate for memory-constrained tenants under skewed workloads.

Problem

• memory underutilization
• cache miss rate
• multi-tenant memory management
• cloud memory provisioning
• stranded memory

Innovation

• memory disaggregation
• RDMA
• miss ratio curve
• multi-tenant memory management
• cloud-scale caching