NEURON-Fabric: CXL-Side Low-Bit Gradient Aggregation for Distributed Training

2026-06-12

AI Summary

This work addresses the substantial memory and communication overhead incurred by gradient All-Reduce in large model distributed training. It presents the first near-memory, low-bitwidth gradient aggregation mechanism implemented on the CXL memory controller, introducing G-Binary sign counting and G-Ternary gated aggregation techniques to construct a 512-bit low-bitwidth data path. The design supports dynamic per-layer selection between FP32 and low-bitwidth paths to balance accuracy and efficiency. Experimental results on CIFAR-10 with ResNet-18 and SST-2 with DistilBERT demonstrate that using low-bitwidth gradients for backbone layers while retaining FP32 for classification heads reduces gradient communication volume to 3.6–5.4% of the baseline, achieving near-FP32 accuracy with at most a 1.67% runtime latency overhead in hardware.

Abstract

In large-model distributed training, especially large language model workloads, gradient All-Reduce increasingly stresses the memory and communication path. This paper asks whether a Compute Express Link (CXL) memory controller can aggregate low-bit gradient signals as gradient cache lines pass through it, while preserving a 32-bit floating-point (FP32) path for workloads, layers, or phases that should not use low-bit approximation. We present NEURON-Fabric, a CXL-side controller architecture that performs packed gradient-binary (G-Binary) sign-count aggregation and gradient-ternary (G-Ternary) gated aggregation near CXL memory, with a control interface for selecting low-bit or FP32 paths. Cycle-level timing experiments show that the measured five-cycle low-bit aggregation datapath adds at most 1.67 percent exposed runtime overhead in the full last-level-cache miss regime; under bandwidth pressure, the same compute stage is hidden by CXL service time. Functional tests confirm byte-exact identity read-back, G-Binary sign-count aggregation, and G-Ternary gating. Training checks quantify the communication and accuracy tradeoff: low-bit aggregation remains close to FP32 on CIFAR-10/ResNet-18 and SST-2/DistilBERT, while full-path low-bit aggregation fails on CIFAR-100/ResNet-18. Layer-aware admission identifies the classifier head as sensitive; keeping the head on FP32 while applying low-bit aggregation to the backbone recovers most accuracy and reduces gradient traffic to 3.6-5.4 percent of the FP32 baseline. Hardware synthesis and FPGA place-and-route estimates suggest that the 512-bit aggregation datapath is small enough to be treated as a near-memory datapath extension rather than a separate accelerator-scale block.

Problem

• distributed training
• gradient aggregation
• low-bit quantization
• CXL memory
• communication overhead

Innovation

• CXL
• low-bit gradient aggregation
• near-memory computing
• gradient compression
• distributed training