Correlations Are Ruining Your Gradient Descent

2024-07-15

arXiv.org

AI Summary

Data correlations induce non-orthogonality in the parameter space after linear transformations across neural network layers, severely degrading the efficiency and stability of gradient descent optimization. This work identifies the covariance structure of intra-layer neuronal responses as a fundamental bottleneck to training performance and establishes— for the first time—a rigorous equivalence between layer-wise dynamic decorrelation across the entire network and the natural gradient optimization objective. Building on this insight, we propose a lightweight intra-layer response decorrelation algorithm based on online whitening and adaptive covariance correction, fully compatible with distributed training and neuromorphic hardware. Experiments demonstrate that our method significantly accelerates convergence of standard backpropagation; more critically, it restores high accuracy and robust convergence to multiple approximate backpropagation algorithms previously rendered infeasible due to accuracy collapse. This enables efficient, low-power deep learning training on brain-inspired hardware, establishing a novel paradigm for neuromorphic AI.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Herein the topics of (natural) gradient descent, data decorrelation, and approximate methods for backpropagation are brought into a common discussion. Natural gradient descent illuminates how gradient vectors, pointing at directions of steepest descent, can be improved by considering the local curvature of loss landscapes. We extend this perspective and show that to fully solve the problem illuminated by natural gradients in neural networks, one must recognise that correlations in the data at any linear transformation, including node responses at every layer of a neural network, cause a non-orthonormal relationship between the model's parameters. To solve this requires a method for decorrelating inputs at each individual layer of a neural network. We describe a range of methods which have been proposed for decorrelation and whitening of node output, and expand on these to provide a novel method specifically useful for distributed computing and computational neuroscience. Implementing decorrelation within multi-layer neural networks, we can show that not only is training via backpropagation sped up significantly but also existing approximations of backpropagation, which have failed catastrophically in the past, benefit significantly in their accuracy and convergence speed. This has the potential to provide a route forward for approximate gradient descent methods which have previously been discarded, training approaches for analogue and neuromorphic hardware, and potentially insights as to the efficacy and utility of decorrelation processes in the brain.

Problem

• Addressing how data correlations impair gradient descent optimization
• Proposing layer-wise decorrelation methods for neural network training
• Improving backpropagation efficiency and approximation accuracy via decorrelation

Innovation

• Layer-wise input decorrelation for neural networks
• Novel method optimized for distributed computing
• Speeds backpropagation and improves approximation accuracy