A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning

2025-11-16

AI Summary

In federated learning, personalized fine-tuning (PFT) often distorts global features due to data heterogeneity, degrading both generalization and personalization performance. To address this, we propose LP-FT—a two-stage fine-tuning strategy: first, linear probing with a frozen backbone and optimized classifier head preserves global feature integrity; second, constrained full-model fine-tuning adapts to local data distributions. We are the first to identify and characterize feature distortion in federated settings, design a staged parameter update mechanism, and theoretically establish LP-FT’s superiority under partial feature overlap and covariate-concept drift. Extensive experiments across seven datasets and six variants demonstrate that LP-FT consistently outperforms state-of-the-art methods—stabilizing global representations, enhancing model robustness, and improving cross-domain adaptability. Our work establishes a novel, theoretically grounded, and practically viable paradigm for personalized federated learning under heterogeneity.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Federated Learning (FL) enables decentralized, privacy-preserving model training but struggles to balance global generalization and local personalization due to non-identical data distributions across clients. Personalized Fine-Tuning (PFT), a popular post-hoc solution, fine-tunes the final global model locally but often overfits to skewed client distributions or fails under domain shifts. We propose adapting Linear Probing followed by full Fine-Tuning (LP-FT), a principled centralized strategy for alleviating feature distortion (Kumar et al., 2022), to the FL setting. Through systematic evaluation across seven datasets and six PFT variants, we demonstrate LP-FT's superiority in balancing personalization and generalization. Our analysis uncovers federated feature distortion, a phenomenon where local fine-tuning destabilizes globally learned features, and theoretically characterizes how LP-FT mitigates this via phased parameter updates. We further establish conditions (e.g., partial feature overlap, covariate-concept shift) under which LP-FT outperforms standard fine-tuning, offering actionable guidelines for deploying robust personalization in FL.

Problem

• Balancing global generalization with local personalization in federated learning
• Addressing overfitting to skewed client data distributions during fine-tuning
• Mitigating federated feature distortion caused by local fine-tuning processes

Innovation

• LP-FT adapts centralized strategy to federated learning
• Phased parameter updates mitigate federated feature distortion
• Balances personalization and generalization across diverse datasets