Revisiting Federated Fine-Tuning: A Single Communication Round is Enough for Foundation Models

2024-12-05

AI Summary

To address the high communication overhead and weak privacy guarantees in federated fine-tuning of large foundation models over distributed data, this paper proposes a novel one-shot aggregation paradigm. Theoretically and empirically, we establish—for the first time—that large models can achieve performance comparable to multi-round federated fine-tuning without iterative parameter aggregation. Our method integrates loss surface analysis with cross-domain distributed optimization, enabling asynchronous local updates and built-in differential privacy enhancement. Evaluated on text generation and text-to-image tasks, our one-shot fine-tuning attains over 98% of the performance of conventional multi-round approaches while reducing communication costs by more than 90%. This significantly improves training efficiency, privacy preservation, and system flexibility—particularly under resource-constrained and heterogeneous federated settings.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Privacy-enhancing technologies
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

The recent advancement of foundation models (FMs) has increased the demand for fine-tuning these models on large-scale cross-domain datasets. To address this, federated fine-tuning has emerged, allowing FMs to be fine-tuned on distributed datasets across multiple devices while ensuring data privacy. However, the substantial parameter size and the multi-round communication in federated learning algorithms result in prohibitively high communication costs, challenging the practicality of federated fine-tuning. In this paper, we identify and analyze, both theoretically and empirically, that the traditional multi-round aggregation algorithms may not be necessary for federated fine-tuning large FMs. Our experiments reveal that a single round of aggregation (i.e., one-shot federated fine-tuning) yields a global model performance comparable to that achieved through multiple rounds of aggregation. Through rigorous mathematical and empirical analyses, we demonstrate that large FMs, due to their extensive parameter sizes and pre-training on general tasks, achieve significantly lower training loss in one-shot federated fine-tuning compared to smaller models. Our extensive experiments show that one-shot federated fine-tuning significantly reduces communication costs. It also has the potential to enable asynchronous aggregation, enhances privacy, and maintains performance consistency with multi-round federated fine-tuning on both text generation and text-to-image generation tasks. Our findings provide insights to revolutionize federated fine-tuning in practice, enhancing efficiency, reducing costs, and expanding accessibility for FMs.

Problem

• Federated fine-tuning faces high communication costs from multi-round aggregation
• Large foundation models require efficient privacy-preserving distributed training methods
• Traditional federated learning approaches may be unnecessary for foundation model fine-tuning

Innovation

• One-shot federated fine-tuning reduces communication rounds
• Single aggregation round maintains model performance consistency
• Asynchronous aggregation enhances privacy and cuts costs