Problem
Research questions and friction points this paper is trying to address.
Stochastic Gradient Descent
High-Probability Convergence
Batched Updates
High-Probability Convergence of SGD via Batched Updates
🤖 AI Summary
本文通过批处理更新方法改进SGD，解决了最后迭代的高概率收敛问题，适用于强凸和非凸目标，并扩展到联邦学习中。
📝 Abstract
Stochastic gradient descent (SGD) is the primary workhorse for large-scale optimization. While the average behavior of its iterates, typically characterized by mean-squared error bounds, is well-understood, obtaining high-probability guarantees for the last iterate remains challenging. Prior approaches to this problem have either imposed restrictive assumptions (such as bounded domains or gradients) or relied on complex proofs involving auxiliary sequences. In this work, we propose Batched SGD, a simple variant that partitions online samples into epochs and performs a single update per epoch using a refined, low-variance gradient estimate. Our main contribution demonstrates that this batching mechanism enables a surprisingly simple high-probability analysis that avoids both restrictive assumptions and auxiliary sequences. Under standard smoothness and norm-sub-Gaussian noise assumptions, we establish near-optimal rates for both strongly convex and non-convex objectives. Furthermore, we show that our batching idea extends naturally to federated learning (FL). We provide the first high-probability guarantees for FL, achieving logarithmic communication complexity, linear speedup in the number of agents, and resilience to data heterogeneity.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Batched SGD
High-Probability Guarantees
Low-Variance Gradient Estimate
Federated Learning
Logarithmic Communication Complexity
🔎 Similar Papers
No similar papers found.
