What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity

2026-07-16

AI Summary

Existing theory struggles to adequately characterize the effectiveness of Local SGD under non-IID data, particularly lacking tight convergence bounds in the general convex setting. This work introduces a bounded second-order heterogeneity assumption and extends it for the first time to general convex objectives. By constructing matching upper and lower bounds, the paper establishes the current best-known nearly tight convergence rate. The analysis further reveals how rare high-curvature clients influence the behavior of serial SGD, thereby substantially deepening the theoretical understanding of Local SGD’s convergence properties.

Abstract

Local SGD, also known as Federated Averaging, is a widely used distributed optimization algorithm. Although Local SGD often outperforms alternatives such as Mini-batch SGD in practice, theory still only partially explains when and why local updates help under realistic data heterogeneity. Recent work by [Patel et al., 2025] shows that a bounded second-order heterogeneity assumption captures the efficiency of Local SGD for strongly convex objectives, and conjectures that the same principle extends to the general convex setting. In this paper, we prove this conjecture by establishing an improved convergence guarantee for Local SGD on general convex objectives under bounded second-order heterogeneity. We also improve the best-known lower bounds for Local SGD in this setting, showing that our upper bounds are nearly tight. Together, these results provide a sharper, more fine-grained convergence theory for Local SGD. As a further application of our techniques, we provide a lower bound for serial SGD with replacement, showing how second-order heterogeneity captures the impact of rare high-curvature clients.

Problem

• Local SGD
• data heterogeneity
• convergence theory
• second-order heterogeneity
• general convex objectives

Innovation

• Local SGD
• bounded second-order heterogeneity
• convergence rates
• general convex optimization
• lower bounds