Orthogonal Weight Modification Enhances Learning Scalability and Convergence Efficiency without Gradient Backpropagation

2026-02-24

AI Summary

This work addresses the high computational cost of traditional backpropagation and the limited efficiency and scalability of existing backpropagation-free learning methods. The authors propose a brain-inspired, low-rank cluster-orthogonal (LOCO) weight perturbation approach that enhances gradient estimation stability through orthogonal constraints, enabling efficient local learning without backpropagation. By integrating low-rank node perturbation, orthogonal weight updates, and a spiking neural network architecture, LOCO is the first method to support deep local training beyond ten layers while maintaining continual learning capabilities. Experiments demonstrate that LOCO converges faster and achieves superior performance compared to existing backpropagation-free algorithms across multiple datasets, with weight updates requiring only O(1) parallel time complexity.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Recognizing the substantial computational cost of backpropagation (BP), non-BP methods have emerged as attractive alternatives for efficient learning on emerging neuromorphic systems. However, existing non-BP approaches still face critical challenges in efficiency and scalability. Inspired by neural representations and dynamic mechanisms in the brain, we propose a perturbation-based approach called LOw-rank Cluster Orthogonal (LOCO) weight modification. We find that low-rank is an inherent property of perturbation-based algorithms. Under this condition, the orthogonality constraint limits the variance of the node perturbation (NP) gradient estimates and enhances the convergence efficiency. Through extensive evaluations on multiple datasets, LOCO demonstrates the capability to locally train the deepest spiking neural networks to date (more than 10 layers), while exhibiting strong continual learning ability, improved convergence efficiency, and better task performance compared to other brain-inspired non-BP algorithms. Notably, LOCO requires only O(1) parallel time complexity for weight updates, which is significantly lower than that of BP methods. This offers a promising direction for achieving high-performance, real-time, and lifelong learning on neuromorphic systems.

Problem

• backpropagation
• learning scalability
• convergence efficiency
• neuromorphic systems
• non-BP methods

Innovation

• orthogonal weight modification
• low-rank perturbation
• node perturbation
• spiking neural networks
• neuromorphic computing