FedOSAA: Improving Federated Learning with One-Step Anderson Acceleration

2025-03-14

AI Summary

First-order federated learning (FL) algorithms suffer from slow convergence and high communication overhead. To address this, we propose integrating a single-step Anderson acceleration into variance-reduced FL frameworks (e.g., FedSVRG, SCAFFOLD), where historical gradient curvature information is fused at the end of local training to approximate the Newton–GMRES direction—marking the first synergistic design of Anderson acceleration and variance reduction in FL. Our method preserves the simplicity and distributed compatibility of first-order algorithms while theoretically guaranteeing local linear convergence. Empirical evaluations across heterogeneous data distributions demonstrate substantial reductions in both communication rounds and computational cost; notably, its convergence speed rivals that of second-order methods such as GIANT, while achieving significant improvements in both communication and computational efficiency.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated learning (FL) is a distributed machine learning approach that enables multiple local clients and a central server to collaboratively train a model while keeping the data on their own devices. First-order methods, particularly those incorporating variance reduction techniques, are the most widely used FL algorithms due to their simple implementation and stable performance. However, these methods tend to be slow and require a large number of communication rounds to reach the global minimizer. We propose FedOSAA, a novel approach that preserves the simplicity of first-order methods while achieving the rapid convergence typically associated with second-order methods. Our approach applies one Anderson acceleration (AA) step following classical local updates based on first-order methods with variance reduction, such as FedSVRG and SCAFFOLD, during local training. This AA step is able to leverage curvature information from the history points and gives a new update that approximates the Newton-GMRES direction, thereby significantly improving the convergence. We establish a local linear convergence rate to the global minimizer of FedOSAA for smooth and strongly convex loss functions. Numerical comparisons show that FedOSAA substantially improves the communication and computation efficiency of the original first-order methods, achieving performance comparable to second-order methods like GIANT.

Problem

• Improves convergence speed in federated learning
• Reduces communication rounds for global optimization
• Enhances efficiency of first-order methods with Anderson acceleration

Innovation

• Combines first-order methods with Anderson acceleration
• Improves convergence using curvature information
• Reduces communication rounds and computation costs