ethernet

Datacenter networks have become a performance bottleneck for data-intensive applications such as machine learning, and existing RDMA protocols struggle to meet the stringent requirements of SmartNICs and accelerators. This paper proposes BALBOA: a service-enhanced RDMA architecture tailored for SmartNICs. Its core is an open-source, customizable RoCE v2–compatible protocol stack capable of supporting hundreds of queue pairs and sustaining 100 Gbps line-rate processing. BALBOA innovatively integrates host-bypass, pipelined compute offloading, and in-network computation, unifying protocol processing and data acceleration on FPGA. It further extends functionality with hardware-accelerated encryption, ML packet classification, and line-rate data preprocessing. Experimental evaluation on an FPGA-based cluster demonstrates that BALBOA achieves latency and throughput comparable to commercial NICs, enabling end-to-end line-rate offloading for recommendation systems. Results validate its high performance, ultra-low latency, and adaptability across diverse application scenarios.

In large-scale distributed machine learning (ML), collective communication suffers from high tail latency, while existing RDMA protocols (e.g., RoCE) exhibit poor scalability due to stringent reliability and in-order delivery guarantees. Method: This paper proposes a lightweight RDMA architecture tailored to ML workloads—intentionally relaxing strict reliability and ordering requirements to exploit ML’s inherent tolerance to partial packet loss. Retransmission and error correction are offloaded to the upper-layer training pipeline; the design employs best-effort transmission, adaptive timeout, priority-based data scheduling, retains DCQCN for congestion control, and integrates Hadamard transform for efficient fault tolerance. Contribution/Results: Experiments show a 2.3× reduction in 99th-percentile latency, 67% reduction in BRAM resource usage, and nearly 2× improvement in NIC fault recovery time—significantly enhancing scalability and resilience of ML communication in ten-thousand-GPU clusters.

This work addresses the high memory overhead and latency in modern data centers caused by the InfiniBand queue-pair abstraction of RDMA, which creates a performance bottleneck at the NIC. The authors present the first clean-room, open-source implementation of Huawei’s Unified Bus (UB) protocol, encompassing both transport and transaction layers. UB decouples application endpoints from host transport state, supports on-demand ordering, and enables remote memory access using native CPU load/store semantics. The project provides reproducible evaluation platforms at RTL, SystemC, and gem5 levels, enabling fair comparison with RoCEv2. Experimental results demonstrate that, for 64-byte remote reads, UB achieves an end-to-end latency of approximately 500 nanoseconds—4.37× lower than RoCEv2—with 2.80× higher throughput, while consuming only 14% of the logic resources on an Alveo U50 FPGA.

Distributed ML at scale (thousands of GPUs) faces a critical bottleneck: tail latency in collective communications. Conventional RDMA solutions (e.g., RoCE) rely heavily on retransmission and strict in-order delivery, resulting in high protocol complexity, tail-latency sensitivity, and pipeline stalls. This work proposes a domain-specific RDMA NIC architecture tailored for ML workloads, the first to explicitly exploit ML’s intrinsic tolerance to packet loss and out-of-order delivery. It eliminates retransmission and rigid ordering, instead adopting adaptive timeout–driven best-effort, out-of-order transmission. Reliability recovery is delegated to the ML layer—via techniques such as Hadamard transforms and erasure coding—enabling a lightweight, resilient NIC design. The architecture maintains full compatibility with standard congestion control mechanisms (DCQCN, EQDS, Swift). Evaluations on Hyperstack and CloudLab demonstrate 2.0× speedup in training time-to-accuracy (TTA), 1.6× higher inference throughput, 3.5× reduction in 99th-percentile latency, 2.7× lower BRAM utilization, and nearly doubled fault tolerance.

In large-scale LLM distributed training, RDMA network failures often cause complete job aborts, while existing application-layer fault tolerance (e.g., checkpointing) incurs disruptive training interruptions. Method: This work proposes, for the first time, an RDMA-layer fine-grained failure state machine and transparent rerouting mechanism that seamlessly redirects RDMA traffic across NICs—without modifying applications (e.g., PyTorch/NCCL) or training code. The solution requires only lightweight driver-level modifications. Contribution/Results: It enables zero-interruption training during faults, reducing progress loss to just 8% until the next checkpoint—a 92% reduction versus conventional approaches—while imposing less than 1.2% overhead on the data path. This bridges a critical gap between network- and application-layer fault tolerance, delivering truly transparent, zero-modification, RDMA-native resilience.

This study addresses the challenge of accurately predicting flow completion time (FCT) for large-scale data transfers in public high-performance wide-area networks (HP-WANs), where limited control over critical path parameters hinders efficient scheduling of compute and storage resources. The work presents the first systematic evaluation of FCT predictability under mainstream TCP congestion control algorithms—including CUBIC, BBRv1, and BBRv3—in real-world HP-WAN environments, augmented with ingress traffic shaping for optimization. Large-scale experiments conducted on the FABRIC testbed demonstrate that, in challenging scenarios characterized by microburst-induced packet loss, BBRv1 combined with traffic shaping significantly enhances FCT predictability, thereby providing reliable latency guarantees for cross-domain data scheduling.

This work addresses the inefficiencies of existing fault-tolerance mechanisms for RDMA link failures, which uniformly retransmit all in-flight requests upon connection disruption, leading to wasted bandwidth, semantic errors, and high recovery overhead. The authors propose a failure-type-aware recovery mechanism that leverages a lightweight completion log to track the execution status of each request, thereby distinguishing between already executed and unexecuted operations. By selectively retransmitting only necessary requests and restoring their results, this approach achieves precise, execution-state-based retransmission for the first time. It prevents redundant execution of non-idempotent operations, ensures transactional consistency, and eliminates the need for connection reestablishment. Experimental results demonstrate negligible memory overhead, a steady-state latency penalty of merely 0.6–10%, and a 65% reduction in recovery retransmission time.

This work addresses the security vulnerabilities of Remote Direct Memory Access (RDMA) in untrusted environments, where its CPU-bypassing nature circumvents conventional security mechanisms. The authors present the first native RDMA encryption implementation directly within the data plane of a programmable Tofino switch, leveraging the P4 language to integrate AES-128 encryption and offload cryptographic operations from the host CPU to the network data plane. This approach ensures strong security without compromising performance. Experimental results demonstrate sustained throughput ranging from 0.37 to 1.9 Gbps for packet sizes between 16 and 128 bytes, confirming the feasibility of simultaneously achieving high performance and robust security. The proposed method overcomes the longstanding challenge of adapting traditional security mechanisms to high-speed RDMA communication.

This work addresses the performance limitations of existing routing schemes in inter-datacenter RDMA networks, which suffer from path asymmetry, delayed congestion signals, and routing conflicts among concurrent flows. To overcome these challenges, the paper proposes LCMP, a novel framework that integrates a unified path-quality scoring mechanism in the control plane with compact congestion signaling in the data plane, enabling low-cost, low-latency, and congestion-aware intelligent multipath scheduling. LCMP further incorporates cost-aware filtering and a diversity-preserving hashing mechanism to mitigate decision conflicts among concurrent flows and enhance path utilization efficiency. Evaluated on eight real-world datacenter testbeds, LCMP reduces median and tail flow completion times by 76% and 64%, respectively, compared to state-of-the-art solutions. Large-scale simulations further confirm its substantial advantages in long-distance scenarios.

This work addresses the dual challenges of 25–35% DRAM waste due to memory fragmentation in cloud environments and high cold-start latency during MicroVM snapshot restoration. To tackle these issues, the authors propose a hierarchical memory pooling architecture that synergistically integrates CXL and RDMA. Snapshot pages are categorized by access热度 (hotness): hot pages reside in a low-latency CXL-attached pool, while cold and zero pages are stored in an RDMA-based remote memory pool. The design includes a novel ownership coherence protocol tailored for non-cache-coherent CXL devices to ensure correctness in multi-host sharing scenarios. Key innovations include the first coordinated use of CXL and RDMA for MicroVM snapshot serving, a hotness-aware snapshot format, and a hybrid loading mechanism featuring hot-page preloading with asynchronous on-demand fetching of cold pages. Experiments demonstrate that the end-to-end restoration latency is reduced by 2.2× on average compared to Firecracker and outperforms the state-of-the-art by 1.1×.