Task-Centric Personalized Federated Fine-Tuning of Language Models

2026-03-30

AI Summary

This work addresses the performance degradation in federated learning caused by task heterogeneity and the limitations of existing personalization methods, which often suffer from poor generalization and severe multi-task interference. To overcome these challenges, the authors propose FedRouter, a task-oriented personalized federated learning framework that shifts the personalization granularity from clients to tasks. FedRouter employs lightweight adapters combined with a two-level clustering mechanism—local sample-level clustering and global cross-client adapter clustering—and dynamically selects the optimal adapter via an evaluation-based routing strategy. Experimental results demonstrate that FedRouter significantly enhances model performance in multi-task federated settings, achieving relative improvements of 6.1% in mitigating task interference and 136% in generalization capability.

Abstract

Federated Learning (FL) has emerged as a promising technique for training language models on distributed and private datasets of diverse tasks. However, aggregating models trained on heterogeneous tasks often degrades the overall performance of individual clients. To address this issue, Personalized FL (pFL) aims to create models tailored for each client's data distribution. Although these approaches improve local performance, they usually lack robustness in two aspects: (i) generalization: when clients must make predictions on unseen tasks, or face changes in their data distributions, and (ii) intra-client tasks interference: when a single client's data contains multiple distributions that may interfere with each other during local training. To tackle these two challenges, we propose FedRouter, a clustering-based pFL that builds specialized models for each task rather than for each client. FedRouter uses adapters to personalize models by employing two clustering mechanisms to associate adapters with specific tasks. A local clustering that associate adapters with task data samples and a global one that associates similar adapters from different clients to construct task-centric personalized models. Additionally, we propose an evaluation router mechanism that routes test samples to the best adapter based on the created clusters. Experiments comparing our method with existing approaches across a multitask dataset, FedRouter demonstrate strong resilience in these challenging scenarios performing up to 6.1% relatively better under tasks interference and up to 136% relative improvement under generalization evaluation.

Problem

• Personalized Federated Learning
• Task Heterogeneity
• Generalization
• Task Interference
• Language Models

Innovation

• Personalized Federated Learning
• Task-Centric Modeling
• Adapter-Based Personalization
• Clustering Mechanism
• Federated Fine-Tuning