Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning

2025-11-05

AI Summary

In federated learning, large-scale model training suffers from high communication overhead and difficulty in simultaneously achieving fast convergence and high accuracy. To address this, we propose ProjFL and its enhanced variant ProjFL+EF. Our core innovation is constructing a low-dimensional shared subspace from historical global descent directions and projecting local gradients onto this subspace before compression and transmission—significantly reducing communication cost. ProjFL supports both unbiased and biased compressors, while ProjFL+EF further incorporates error feedback to ensure convergence. We provide rigorous theoretical guarantees for convergence under strongly convex, convex, and non-convex objectives. Extensive experiments on standard image classification benchmarks demonstrate that our methods reduce communication costs by up to 90% while maintaining test accuracy comparable to state-of-the-art baselines.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Federated Learning (FL) enables decentralized model training across multiple clients while optionally preserving data privacy. However, communication efficiency remains a critical bottleneck, particularly for large-scale models. In this work, we introduce two complementary algorithms: ProjFL, designed for unbiased compressors, and ProjFL+EF, tailored for biased compressors through an Error Feedback mechanism. Both methods rely on projecting local gradients onto a shared client-server subspace spanned by historical descent directions, enabling efficient information exchange with minimal communication overhead. We establish convergence guarantees for both algorithms under strongly convex, convex, and non-convex settings. Empirical evaluations on standard FL classification benchmarks with deep neural networks show that ProjFL and ProjFL+EF achieve accuracy comparable to existing baselines while substantially reducing communication costs.

Problem

• Addressing communication bottlenecks in federated learning systems
• Reducing communication overhead while maintaining model accuracy
• Developing efficient gradient projection methods for decentralized training

Innovation

• Projects gradients onto historical descent directions
• Uses ProjFL for unbiased compression algorithms
• Incorporates Error Feedback for biased compressors