Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity

2026-06-19

AI Summary

This work addresses the limitations of traditional deep neural networks, which rely on backpropagation and suffer from inter-layer update locking and high communication overhead, thereby restricting training parallelism. The authors propose a Hierarchical Block Local Learning (HBLL) framework that decomposes the network into hierarchically connected blocks and formulates local learning objectives grounded in variational principles, enabling effective information propagation without end-to-end backpropagation. This approach achieves, for the first time, training of deep networks with $\mathcal{O}(\log N)$ parallel time complexity and implicitly constructs multi-path subnetworks to support flexible inference. HBLL is applicable to both feedforward and recurrent architectures and demonstrates competitive performance across vision, language, and sequence modeling tasks.

Abstract

Modern deep neural network architectures are trained via backpropagation, which requires errors to be sequentially propagated through all layers before parameters can be updated. This introduces two limitations: locking, where layer-wise updates are strictly interdependent and cannot proceed in parallel, and the weight transport problem, which requires symmetric forward and backward pathways for exact gradient computation. These constraints restrict parallelism, increase memory and communication overhead, and pose challenges for scalable learning. In this work, we propose Hierarchical Block-Local Learning (HBLL), a framework that decomposes deep neural networks into hierarchically linked blocks trained using local learning objectives derived from variational principles, eliminating the need for full end-to-end backpropagation while maintaining effective information propagation across the network. HBLL is the first algorithm that is able to train deep neural networks in $\mathcal{O}(\log N)$ parallel time complexity, where $N$ is the number of network layers. We show that HBLL implicitly defines a family of subnetworks corresponding to different hierarchical paths, enabling flexible inference with different effective numbers of layers. We evaluate HBLL on a set of challenging vision and language modeling tasks, achieving competitive performance. We also extend HBLL to recurrent sequence architectures, applying to settings that otherwise rely on backpropagation through time.

Problem

• backpropagation
• locking
• weight transport problem
• parallelism
• scalable learning

Innovation

• Hierarchical Block-Local Learning
• O(log N) parallel time complexity
• local learning
• backpropagation-free training
• weight transport problem