FedGMR: Federated Learning with Gradual Model Restoration under Asynchrony and Model Heterogeneity

2025-12-04

AI Summary

To address slow convergence and poor generalization in heterogeneous federated learning caused by insufficient submodel capacity on bandwidth-constrained clients (BCCs), this paper proposes FedGMR. The framework introduces a progressive model recovery mechanism that dynamically increases the density of client submodels, and a mask-aware asynchronous aggregation rule whose theoretical analysis shows that aggregation error decays with increasing submodel density and convergence approaches that of full-model FL. By integrating progressive expansion, mask-aware aggregation, and asynchronous communication, FedGMR is evaluated on FEMNIST, CIFAR-10, and ImageNet-100. Experiments demonstrate that FedGMR significantly accelerates convergence and improves final accuracy—particularly under high data heterogeneity, strong Non-IIDness, and severe resource constraints—while maintaining communication efficiency and model adaptability.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Federated learning (FL) holds strong potential for distributed machine learning, but in heterogeneous environments, Bandwidth-Constrained Clients (BCCs) often struggle to participate effectively due to limited communication capacity. Their small sub-models learn quickly at first but become under-parameterized in later stages, leading to slow convergence and degraded generalization. We propose FedGMR - Federated Learning with Gradual Model Restoration under Asynchrony and Model Heterogeneity. FedGMR progressively increases each client's sub-model density during training, enabling BCCs to remain effective contributors throughout the process. In addition, we develop a mask-aware aggregation rule tailored for asynchronous MHFL and provide convergence guarantees showing that aggregated error scales with the average sub-model density across clients and rounds, while GMR provably shrinks this gap toward full-model FL. Extensive experiments on FEMNIST, CIFAR-10, and ImageNet-100 demonstrate that FedGMR achieves faster convergence and higher accuracy, especially under high heterogeneity and non-IID settings.

Problem

• Addresses slow convergence in federated learning with bandwidth-constrained clients
• Mitigates degraded generalization due to under-parameterized sub-models in later stages
• Enables effective participation of heterogeneous clients under asynchronous conditions

Innovation

• Gradual model restoration for bandwidth-constrained clients
• Mask-aware aggregation for asynchronous heterogeneous federated learning
• Convergence guarantees scaling with average sub-model density