NEURON-Fabric: Architecture-Runtime Co-Design for Controlled Low-Bit Gradient Communication

2026-06-24

AI Summary

This work addresses the high communication overhead in large-scale neural network training, where static low-bit gradient compression often leads to instability due to variations in model architecture or training phase. The authors propose NEURON-Fabric, which frames low-bit gradient communication as a co-design problem between system architecture and runtime execution. By integrating calibration-driven dynamic control, model-aware communication hooks, sign-count compact aggregation, and capacity-aware fallback mechanisms, NEURON-Fabric preserves semantic fidelity within mixed-precision buckets and enables unified routing across diverse model families. Experiments on vision models, Transformers, and billion-parameter language models demonstrate that the method substantially reduces communication volume while maintaining training accuracy close to that of FP32 baselines.

Abstract

Large-scale neural-network training repeatedly aggregates gradients across devices, making communication a central cost in distributed learning. Low-bit gradient aggregation can reduce this cost, but applying it as a static replacement for full-precision communication can destabilize training because safe precision depends on training phase, model structure, runtime bucketization, and the communication substrate. This paper presents NEURON-Fabric, a profile-guided runtime system for controlled low-bit gradient communication. NEURON-Fabric uses calibrated operating profiles, model-aware runtime bindings, online training-health monitoring, and reducer-capacity checks to decide when low-bit aggregation should be admitted, when execution should fall back to FP32, and which model regions are eligible for each route. The runtime preserves model semantics inside mixed DDP buckets and treats reducer admission as an architecture-runtime co-design problem rather than as a standalone compression operator. Across vision, Transformer, and autoregressive language-model workloads, NEURON-Fabric validates the path from calibration to distributed communication-hook execution. Static low-bit communication can collapse training accuracy, while profile-guided control preserves accuracy near full-precision references or calibrated targets and reduces modeled gradient-communication traffic in the evaluated settings. Transformer and billion-parameter language-model checks show that the same routing and fallback mechanisms execute across model families and multi-node deployments. Reducer-side replay and reducer-path measurements identify when compact sign-count aggregation is expected to reduce communication cost and when endpoint capacity should trigger fallback.

Problem

• low-bit gradient communication
• distributed training
• training stability
• gradient aggregation
• communication cost

Innovation

• low-bit gradient communication
• architecture-runtime co-design
• profile-guided runtime
• distributed training
• adaptive precision control