Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learning

2026-05-04

AI Summary

This work addresses the challenge in long-tailed personalized federated learning where fine-tuning disrupts the intrinsic class balance of the global model, often degrading performance below that of a zero-shot baseline and propagating this bias to local models. To mitigate this issue, the authors propose FedPuReL, a novel approach that preserves the global model’s balanced knowledge through zero-shot prediction-guided gradient purification and achieves unbiased personalization by introducing residual correction atop a frozen global model. FedPuReL is the first to elucidate the mechanism by which fine-tuning undermines class balance and pioneers a new paradigm that integrates gradient purification with residual learning. Extensive experiments demonstrate that FedPuReL consistently outperforms existing methods across diverse long-tailed settings, achieving state-of-the-art results in both global and personalized performance.

Abstract

Personalized federated learning (PFL) with foundation models has emerged as a promising paradigm enabling clients to adapt to heterogeneous data distributions. However, real-world scenarios often face the co-occurrence of non-IID data and long-tailed class distributions, presenting unique challenges that remain underexplored in PFL. In this paper, we investigate this long-tailed personalized federated learning and observe that current methods suffer from two limitations: (i) fine-tuning degrades performance below zero-shot baselines due to the erosion of inherent class balance in foundation models; (ii) conventional personalization techniques further transfer this bias to local models through parameter or feature-level fusion. To address these challenges, we propose Federated Learning via Gradient Purification and Residual Learning (FedPuReL), which preserves balanced knowledge in the global model while enabling unbiased personalization. Specifically, we purify local gradients using zero-shot predictions to maintain a class-balanced global model, and model personalization as residual correction atop the frozen global model. Extensive experiments demonstrate that FedPuReL consistently outperforms state-of-the-art methods, achieving superior performance on both global and personalized models across diverse long-tailed scenarios. The code is available at https://github.com/shihaohou/FedPuReL.

Problem

• personalized federated learning
• long-tailed
• foundation models
• class imbalance
• fine-tuning

Innovation

• Personalized Federated Learning
• Long-tailed Learning
• Foundation Models
• Gradient Purification
• Residual Learning