Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression

2025-01-23

AI Summary

This work addresses the high communication overhead and slow convergence in heterogeneous distributed logistic regression. We propose and theoretically analyze the Local Gradient Descent (Local GD) algorithm. Our key contribution is the first proof that, under a large step size (η ≫ 1/K), Local GD achieves a convergence rate of O(1/KR)—breaking the conventional Ω(1/R) bottleneck. This acceleration stems from a genuine benefit of increasing the number K of local updates, overturning the standard analytical paradigm requiring η ≤ 1/K. By unifying nonconvex and strongly convex analysis frameworks and explicitly modeling data heterogeneity, we establish the first rigorous convergence theory that jointly optimizes communication efficiency and statistical heterogeneity in federated learning. Our analysis yields the tightest known theoretical convergence bound for distributed logistic regression under heterogeneous data settings.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Local Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We analyze two variants of Local Gradient Descent applied to distributed logistic regression with heterogeneous, separable data and show convergence at the rate $O(1/KR)$ for $K$ local steps and sufficiently large $R$ communication rounds. In contrast, all existing convergence guarantees for Local GD applied to any problem are at least $Omega(1/R)$, meaning they fail to show the benefit of local updates. The key to our improved guarantee is showing progress on the logistic regression objective when using a large stepsize $eta gg 1/K$, whereas prior analysis depends on $eta leq 1/K$.

Problem

• Distributed Computing
• Logistic Regression
• Gradient Descent Optimization

Innovation

• Distributed Logistic Regression
• Accelerated Convergence
• Large Step Size Optimization