Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity

2025-06-01

AI Summary

In low-data, highly heterogeneous federated learning, the global model’s intermediate layers often stagnate early in training—termed the “inter-layer inertia phenomenon”—leading to ineffective aggregation and degraded client collaboration. This paper presents the first systematic analysis and formal modeling of this issue. We propose LIPS, a lightweight, plug-and-play method that integrates gradient sparsification with layer-adaptive masking into standard FedAvg. By periodically injecting transient sparsity without additional parameters or scheduling overhead, LIPS reactivates intermediate-layer updates. Evaluated across CIFAR-10/100 and Tiny-ImageNet under diverse network architectures and heterogeneity settings, LIPS consistently improves test accuracy by an average of 3.2%, while significantly enhancing aggregation efficacy and convergence stability.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated learning (FL) enables collaborative model training across decentralized clients while preserving data privacy, leveraging aggregated updates to build robust global models. However, this training paradigm faces significant challenges due to data heterogeneity and limited local datasets, which often impede effective collaboration. In such scenarios, we identify the Layer-wise Inertia Phenomenon in FL, wherein the middle layers of global model undergo minimal updates after early communication rounds, ultimately limiting the effectiveness of global aggregation. We demonstrate the presence of this phenomenon across a wide range of federated settings, spanning diverse datasets and architectures. To address this issue, we propose LIPS (Layer-wise Inertia Phenomenon with Sparsity), a simple yet effective method that periodically introduces transient sparsity to stimulate meaningful updates and empower global aggregation. Experiments demonstrate that LIPS effectively mitigates layer-wise inertia, enhances aggregation effectiveness, and improves overall performance in various FL scenarios. This work not only deepens the understanding of layer-wise learning dynamics in FL but also paves the way for more effective collaboration strategies in resource-constrained environments. Our code is publicly available at: https://github.com/QiaoXiao7282/LIPS.

Problem

• Overcoming data heterogeneity in federated learning
• Addressing limited local datasets in FL collaboration
• Mitigating layer-wise inertia in global model updates

Innovation

• Introduces transient sparsity to stimulate updates
• Mitigates layer-wise inertia in federated learning
• Enhances global aggregation effectiveness