Flexible Personalized Split Federated Learning for On-Device Fine-Tuning of Foundation Models

2025-08-14

AI Summary

To address challenges in personalizing foundation models for downstream tasks—namely, scarce client-side data, statistical heterogeneity, and constrained computational resources—this paper proposes a resource-adaptive personalized federated learning framework. The framework innovatively integrates split learning with personalized federated learning, enabling clients to dynamically partition the model according to local compute capacity and collaboratively train their respective segments. A gradient alignment strategy is introduced to facilitate efficient personalized adaptation atop the global model. Crucially, raw data remain on-device, eliminating data transmission and substantially reducing both communication and computation overhead. Extensive experiments demonstrate that the method achieves superior personalization efficiency and final accuracy compared to mainstream baselines (e.g., FedAvg, pFedMe), particularly under low-resource and highly heterogeneous settings. Under Non-IID data distributions, it yields absolute accuracy improvements of 3.2–5.7 percentage points.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Fine-tuning foundation models is critical for superior performance on personalized downstream tasks, compared to using pre-trained models. Collaborative learning can leverage local clients' datasets for fine-tuning, but limited client data and heterogeneous data distributions hinder effective collaboration. To address the challenge, we propose a flexible personalized federated learning paradigm that enables clients to engage in collaborative learning while maintaining personalized objectives. Given the limited and heterogeneous computational resources available on clients, we introduce extbf{flexible personalized split federated learning (FlexP-SFL)}. Based on split learning, FlexP-SFL allows each client to train a portion of the model locally while offloading the rest to a server, according to resource constraints. Additionally, we propose an alignment strategy to improve personalized model performance on global data. Experimental results show that FlexP-SFL outperforms baseline models in personalized fine-tuning efficiency and final accuracy.

Problem

• Addresses limited client data in collaborative foundation model fine-tuning
• Solves heterogeneous data distribution challenges in federated learning
• Optimizes resource-constrained on-device model training via flexible splitting

Innovation

• Flexible personalized split federated learning paradigm
• Resource-adaptive local and server model splitting
• Alignment strategy for global data performance