Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysis

2025-01-08

AI Summary

This paper investigates the convergence advantages of LocalSGD and SCAFFOLD over standard minibatch SGD (MbSGD) in distributed stochastic optimization. Addressing the limitation of existing theoretical analyses—which rely on strong assumptions such as gradient similarity or quadratic objectives—the work establishes the first rigorous convergence acceleration guarantees under more realistic, relaxed conditions: weak convexity and Hessian continuity. Methodologically, it conducts a fine-grained analysis of higher-order gradient and Hessian similarity to characterize algorithmic behavior. The contributions are threefold: (1) It proves that LocalSGD achieves acceleration over MbSGD without requiring strong gradient similarity, identifying Hessian similarity as the key enabler of its speedup; (2) it extends SCAFFOLD’s acceleration guarantees to general non-quadratic, weakly convex objectives; and (3) it provides a solid theoretical foundation for federated learning and other heterogeneous distributed settings where data and model heterogeneity are prevalent.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

LocalSGD and SCAFFOLD are widely used methods in distributed stochastic optimization, with numerous applications in machine learning, large-scale data processing, and federated learning. However, rigorously establishing their theoretical advantages over simpler methods, such as minibatch SGD (MbSGD), has proven challenging, as existing analyses often rely on strong assumptions, unrealistic premises, or overly restrictive scenarios. In this work, we revisit the convergence properties of LocalSGD and SCAFFOLD under a variety of existing or weaker conditions, including gradient similarity, Hessian similarity, weak convexity, and Lipschitz continuity of the Hessian. Our analysis shows that (i) LocalSGD achieves faster convergence compared to MbSGD for weakly convex functions without requiring stronger gradient similarity assumptions; (ii) LocalSGD benefits significantly from higher-order similarity and smoothness; and (iii) SCAFFOLD demonstrates faster convergence than MbSGD for a broader class of non-quadratic functions. These theoretical insights provide a clearer understanding of the conditions under which LocalSGD and SCAFFOLD outperform MbSGD.

Problem

• LocalSGD
• SCAFFOLD
• MbSGD

Innovation

• LocalSGD
• SCAFFOLD
• convergence performance