Worker Disagreement Reveals Sharp Directions in Local SGD

2026-05-26

AI Summary

This work addresses the high anisotropy of loss landscapes in deep neural networks, where explicitly computing the Hessian to identify dominant sharp directions is computationally prohibitive. The authors propose leveraging the covariance of the averaging gaps among workers in Local SGD to implicitly uncover the dominant Hessian subspace of the loss function. They provide the first theoretical guarantee that worker disagreement serves as a low-cost, Hessian-free estimator capable of effectively capturing the primary components of the gradient within this subspace. Built upon stochastic gradient noise analysis and a principled covariance–curvature relationship, the method efficiently approximates dominant Hessian directions across diverse architectures—including MLPs, CNNs, and Transformers—while substantially reducing computational overhead.

Abstract

Deep neural network training often exhibits highly anisotropic loss geometry, where a few sharp dominant Hessian directions coexist with a large flatter bulk. Gradients tend to align disproportionately with these dominant directions, although stable progress often requires movement through flatter bulk directions. Estimating the dominant subspace is therefore useful but costly with direct Hessian-based methods. We show that standard Local SGD exposes this geometry through worker disagreement. We theoretically show that the worker-average gap covariance is shaped by stochastic-gradient noise and Hessian curvature, causing workers to disagree along sharp, curvature-sensitive directions. Thus, worker-average gaps provide a cheap Hessian-free estimator of the dominant subspace. Experiments on MLPs, CNNs, and Transformers show that subspaces formed by worker-average gaps capture a substantial fraction of the gradient component lying in the dominant Hessian eigenspace.

Problem

• anisotropic loss geometry
• dominant Hessian directions
• worker disagreement
• Local SGD
• subspace estimation

Innovation

• Local SGD
• Hessian curvature
• worker disagreement
• loss landscape anisotropy
• dominant subspace estimation