Federated Customization of Large Models: Approaches, Experiments, and Insights

2026-01-02

IEEE Network

AI Summary

This work addresses the challenges of data privacy and communication efficiency in customizing large models within federated learning. The authors systematically evaluate the suitability of various parameter-efficient fine-tuning methods and, for the first time, introduce prefix-tuning into the federated learning framework, proposing Federated Prefix-Tuning. This approach achieves model performance comparable to centralized training while significantly improving communication efficiency and robustness. Experimental results across multiple tasks demonstrate that the proposed method either outperforms or matches existing federated customization strategies—including full fine-tuning, other parameter-efficient fine-tuning techniques, and knowledge distillation—thereby validating its effectiveness and practicality.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.

Problem

• federated learning
• large model customization
• prefix-tuning
• model personalization
• distributed optimization

Innovation

• federated learning
• prefix-tuning
• large model customization
• federated customization
• efficient fine-tuning