Lazy But Effective: Collaborative Personalized Federated Learning with Heterogeneous Data

2025-05-05

AI Summary

Non-IID data across clients in federated learning impedes the global model’s ability to achieve satisfactory personalized performance. To address this, we propose a distributed client clustering and collaborative training framework for personalized federated learning. Our core contribution is a lightweight “Lazy Influence” influence function approximation, enabling decentralized, label-free semantic similarity–based client clustering without access to centralized data or labels. Coupled with clustering-aware personalized model aggregation and local fine-tuning, the framework achieves efficient non-IID optimization. Evaluated on Nordic language prediction and CIFAR-100, our method fully recovers the global model’s performance loss under non-IID conditions, matching the performance of oracle clustering and outperforming mainstream baselines by an average of 17%.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Applications of semantic technologies for improving search, browsing, recommendation, personalization

Abstract

In Federated Learning, heterogeneity in client data distributions often means that a single global model does not have the best performance for individual clients. Consider for example training a next-word prediction model for keyboards: user-specific language patterns due to demographics (dialect, age, etc.), language proficiency, and writing style result in a highly non-IID dataset across clients. Other examples are medical images taken with different machines, or driving data from different vehicle types. To address this, we propose a simple yet effective personalized federated learning framework (pFedLIA) that utilizes a computationally efficient influence approximation, called `Lazy Influence', to cluster clients in a distributed manner before model aggregation. Within each cluster, data owners collaborate to jointly train a model that captures the specific data patterns of the clients. Our method has been shown to successfully recover the global model's performance drop due to the non-IID-ness in various synthetic and real-world settings, specifically a next-word prediction task on the Nordic languages as well as several benchmark tasks. It matches the performance of a hypothetical Oracle clustering, and significantly improves on existing baselines, e.g., an improvement of 17% on CIFAR100.

Problem

• Addressing performance drop in global models due to non-IID client data
• Enhancing personalized federated learning with efficient client clustering
• Improving model accuracy for heterogeneous data distributions

Innovation

• Distributed client clustering using Lazy Influence
• Collaborative model training within clusters
• Efficient personalized federated learning framework