Accelerated Training through Iterative Gradient Propagation Along the Residual Path
The inherent sequentiality of backpropagation in deep models severely limits training efficiency, particularly in residual architectures such as ResNet, Transformer, and RNN. To address this, we propose Highway-BP—a novel algorithm that, for the first time, decomposes gradients along residual connections into a multi-path additive form, enabling approximate backpropagation via iterative gradient accumulation and inter-layer parallel backward passes. Highway-BP requires no modification to the forward pass and is natively compatible with standard residual models. Theoretical analysis guarantees gradient consistency, and an approximation-optimization strategy is introduced to balance accuracy and efficiency. Experiments across image classification and machine translation demonstrate that Highway-BP achieves 1.8–2.4× speedup in training time while preserving model accuracy (accuracy degradation <0.3%), effectively alleviating the backpropagation bottleneck in deep networks.