Constant Stepsize Local GD for Logistic Regression: Acceleration by Instability

2025-06-16

AI Summary

This paper studies the convergence of Local Gradient Descent (Local GD) for logistic regression under heterogeneous data, breaking the classical step-size constraint η ≤ 1/K (where K is the communication interval) and allowing arbitrary η > 0. Methodologically, we construct a novel corrected Lyapunov function to characterize a two-phase dynamical behavior: an initial unstable phase followed by a stable phase. We establish, for the first time, that constant-step-size Local GD achieves an O(1/ηKR) convergence rate in the separable heterogeneous setting—strictly improving upon the generic O(1/R) bound for smooth convex objectives. A key contribution is the identification of a new acceleration mechanism: heterogeneity-induced instability from local updates itself accelerates convergence, distinct from instability arising from large step sizes in centralized settings. The initial unstable phase lasts only Õ(ηKM) communication rounds, significantly enhancing overall efficiency.

Technology Category

• Search and Optimization: Local Search
• Machine Learning: Learning with Manifolds
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing analysis of Local (Stochastic) Gradient Descent for heterogeneous objectives requires stepsizes $eta leq 1/K$ where $K$ is the communication interval, which ensures monotonic decrease of the objective. In contrast, we analyze Local Gradient Descent for logistic regression with separable, heterogeneous data using any stepsize $eta>0$. With $R$ communication rounds and $M$ clients, we show convergence at a rate $mathcal{O}(1/eta K R)$ after an initial unstable phase lasting for $widetilde{mathcal{O}}(eta K M)$ rounds. This improves upon the existing $mathcal{O}(1/R)$ rate for general smooth, convex objectives. Our analysis parallels the single machine analysis of~cite{wu2024large} in which instability is caused by extremely large stepsizes, but in our setting another source of instability is large local updates with heterogeneous objectives.

Problem

• Analyzing Local Gradient Descent for logistic regression with heterogeneous data
• Studying convergence under large stepsizes and heterogeneous objectives
• Improving convergence rate compared to general smooth convex objectives

Innovation

• Local Gradient Descent with any stepsize
• Convergence after initial unstable phase
• Improved rate for heterogeneous objectives