FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection

2026-04-26

AI Summary

This work addresses the challenges of slow convergence and high communication and memory overhead in federated learning under heterogeneous and resource-constrained settings. We propose a federated optimization method that integrates stochastic low-rank subspace projection with momentum mechanisms. By applying low-rank compression to client-side gradient updates, our approach significantly reduces communication volume and memory footprint while preserving model accuracy. Leveraging non-convex optimization theory, we provide rigorous convergence guarantees for the proposed algorithm. Experimental results on heterogeneous MNIST tasks demonstrate that our method achieves accuracy comparable to or better than FedAvg and existing sparsity- or low-rank-based baselines, while substantially lowering both communication and memory costs.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated learning enables a population of clients to collaboratively train machine learning models without exchanging their raw data, but standard algorithms such as FedAvg suffer from slow convergence and high communication and memory costs in heterogeneous, resource-constrained environments. We introduce FedSLoP, a federated optimization algorithm that combines stochastic low-rank subspace projections of gradients, thereby reducing the dimension of communicated and stored updates while preserving optimization progress. On the theoretical side, we develop a detailed nonconvex convergence analysis under standard smoothness and bounded-variance assumptions, showing that FedSLoP is guaranteed to converge to a first-order stationary point at a rate of $O(1/\sqrt{NT})$. On the empirical side, we conduct extensive experiments on federated MNIST classification with heterogeneous data partitions, showing that FedSLoP substantially reduces communication volume and client-side memory while achieving competitive or better accuracy compared with FedAvg and representative sparse or low-rank baselines. Together, our results demonstrate that random subspace momentum methods such as FedSLoP provide a principled and effective approach to communication- and memory-efficient federated learning. Codes are available at: https://github.com/pkumelon/FedSLoP.git.

Problem

• federated learning
• communication efficiency
• memory efficiency
• heterogeneous environments
• gradient compression

Innovation

• federated learning
• low-rank gradient projection
• memory efficiency
• communication compression
• nonconvex convergence