Cross-IP Request Coalescing: Relocating the Fan-out Point in Virtualized I/O

2026-07-03

AI Summary

This work addresses the significant performance degradation in virtualized I/O for AI workloads, where a single user request traversing multiple IP modules incurs multiple round trips and context switches. To mitigate this overhead, the authors propose merging multi-device I/O requests into a single composite request on the guest side and uniformly decomposing and dispatching them within an SPDK vhost-user backend. This approach achieves cross-IP request coalescing for the first time in a kernel-bypass, user-space storage framework, eliminating the inherent per-device submission overhead and fundamentally restructuring the virtualized I/O path. Experimental results in an SPDK virtualization environment demonstrate up to a 78% reduction in latency, with performance gains becoming increasingly pronounced as concurrency scales.

Abstract

Cloud data centers rely on virtualization technologies to serve AI workloads in multi-tenant environments. With the growing scale of data-intensive AI workloads, the performance of storage I/O paths at the virtualization layer has become a critical factor. A single user request often crosses multiple IP blocks, where functional units such as storage, GPU, and accelerator devices under virtualization fan out into separate stack traversals between the guest and the backend. As a result, round-trip and context-switching overheads accumulate with the number of devices. In this letter, we identify that a dominant factor in this overhead lies not in the kernel-mediated I/O path alone, but in the per-device submission structure itself, which persists even in user-space, kernel-bypass storage frameworks such as SPDK. To address this, we propose cross-IP request coalescing, which relocates the fan-out point from the guest to the SPDK vhost-user backend. The guest submits multi-device I/O as a single compound request, and the accelerated bdev at the backend decomposes and dispatches it to each target device, replacing multiple per-device guest-backend round trips with a single submission. Evaluation in an SPDK-based virtualized environment shows that the proposed approach achieves up to 1.78x lower latency than the per-device baseline, with the benefit growing as concurrency increases.

Problem

• virtualized I/O
• request coalescing
• multi-device I/O
• latency overhead
• SPDK

Innovation

• Cross-IP Request Coalescing
• Virtualized I/O
• SPDK
• Fan-out Point Relocation
• Multi-device I/O