What Makes Local Updates Effective: The Role of Data Heterogeneity and Smoothness

2025-06-30

AI Summary

This work investigates the theoretical efficacy of local-update algorithms—particularly Local SGD—in distributed and federated optimization under data heterogeneity, a prevalent challenge in practical non-IID settings. Method: We propose a fine-grained analytical framework grounded in consensus error, introduce a novel third-order smoothness condition and relaxed heterogeneity modeling, and establish the necessity and sufficiency of second-order heterogeneity assumptions. Contributions/Results: Our analysis delivers tight convergence bounds unifying convex and non-convex objectives, and identifies precise conditions under which Local SGD strictly outperforms centralized SGD or mini-batch methods. We further extend the framework to online federated learning, deriving foundational regret bounds for both first-order and bandit feedback settings. These results yield minimax complexity characterizations across multiple problem classes, providing the first self-consistent, verifiable theoretical guideline for local-update algorithms in heterogeneous environments.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

This thesis contributes to the theoretical understanding of local update algorithms, especially Local SGD, in distributed and federated optimization under realistic models of data heterogeneity. A central focus is on the bounded second-order heterogeneity assumption, which is shown to be both necessary and sufficient for local updates to outperform centralized or mini-batch methods in convex and non-convex settings. The thesis establishes tight upper and lower bounds in several regimes for various local update algorithms and characterizes the min-max complexity of multiple problem classes. At its core is a fine-grained consensus-error-based analysis framework that yields sharper finite-time convergence bounds under third-order smoothness and relaxed heterogeneity assumptions. The thesis also extends to online federated learning, providing fundamental regret bounds under both first-order and bandit feedback. Together, these results clarify when and why local updates offer provable advantages, and the thesis serves as a self-contained guide for analyzing Local SGD in heterogeneous environments.

Problem

• Understanding Local SGD in distributed optimization with data heterogeneity
• Establishing bounds for local update algorithms in various regimes
• Analyzing online federated learning with regret bounds under feedback

Innovation

• Bounded second-order heterogeneity assumption analysis
• Fine-grained consensus-error-based analysis framework
• Online federated learning regret bounds