Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning

2024-12-10

arXiv.org

AI Summary

To address inefficient global knowledge injection and excessive local model personalization—leading to degraded generalization—in personalized federated learning (pFL), this paper proposes FLAYER, the first layer-aware pFL framework. Its core contributions are: (1) layer-adaptive initialization, enabling on-demand injection of global knowledge into specific layers; (2) dynamic layered learning rate scheduling, jointly optimizing personalization and generalization; and (3) selective layered parameter upload and aggregation, enhancing preservation of global representations. Evaluated on four non-IID benchmark datasets spanning computer vision and natural language processing, FLAYER achieves an average inference accuracy improvement of 5.40% (up to +14.29%) over six state-of-the-art methods. It delivers superior personalized modeling performance while maintaining low communication and computational overhead.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Real-life deployment of federated Learning (FL) often faces non-IID data, which leads to poor accuracy and slow convergence. Personalized FL (pFL) tackles these issues by tailoring local models to individual data sources and using weighted aggregation methods for client-specific learning. However, existing pFL methods often fail to provide each local model with global knowledge on demand while maintaining low computational overhead. Additionally, local models tend to over-personalize their data during the training process, potentially dropping previously acquired global information. We propose FLAYER, a novel layer-wise learning method for pFL that optimizes local model personalization performance. FLAYER considers the different roles and learning abilities of neural network layers of individual local models. It incorporates global information for each local model as needed to initialize the local model cost-effectively. It then dynamically adjusts learning rates for each layer during local training, optimizing the personalized learning process for each local model while preserving global knowledge. Additionally, to enhance global representation in pFL, FLAYER selectively uploads parameters for global aggregation in a layer-wise manner. We evaluate FLAYER on four representative datasets in computer vision and natural language processing domains. Compared to six state-of-the-art pFL methods, FLAYER improves the inference accuracy, on average, by 5.40% (up to 14.29%).

Problem

• Optimizing personalized federated learning for non-IID data challenges
• Balancing local model personalization with global knowledge retention
• Reducing computational overhead in adaptive layer-wise learning methods

Innovation

• Adaptive layer-wise learning for pFL optimization
• Dynamic learning rate adjustment per layer
• Selective layer-wise parameter upload for aggregation