Scaling Synthetic-Image Pre-Training for Federated Fine-Tuning of Large Vision Models

2026-07-14

AI Summary

This work addresses three key challenges in federated fine-tuning—resource constraints, system heterogeneity, and non-independent and identically distributed (non-IID) data—by proposing FeDiSyn, a unified framework that jointly optimizes synthetic image pretraining and federated fine-tuning. FeDiSyn innovatively integrates device-aware diffusion models to generate distribution-aligned synthetic data, a contribution-aware LoRA mechanism for parameter-efficient fine-tuning, and a dynamic bandwidth allocation strategy. Notably, it establishes the first scaling law for pretrained models in federated fine-tuning. Experimental results on real-world platforms demonstrate that FeDiSyn reduces training time by over 52.5% and communication overhead by 97.2%, all while preserving model accuracy.

Abstract

Federated fine-tuning (FedFT) enables adapting pre-trained large vision models (LVMs) on distributed, privacy-sensitive devices, while its practical deployment is hindered by three critical challenges: resource constraints, system heterogeneity, and non-IID data. While prior studies partially address these issues, e.g., by pre-training initial models on synthetic images to mitigate the adverse effects of non-IID data, or leveraging parameter-efficient fine-tuning (PEFT) methods like low-rank adaptation (LoRA) to reduce resource consumption, they remain inadequate and fragmented. Specifically, existing synthetic image generation methods fail to capture device-specific feature distributions, while current PEFT-based FedFT methods often undervalue weaker devices that may provide critical information. More importantly, the separate optimization of pre-training and FedFT neglects their inherent connection, lacking a holistic perspective to maximize training efficiency. To overcome these limitations, we propose FeDiSyn, a unified framework that holistically considers the interplay between pre-training and FedFT to minimize the overall LVM training time. Specifically, FeDiSyn introduces: (i) a scaling law for FedFT pre-training to determine the optimal number of synthetic images, balancing pre-training benefit against generation/pre-training cost, (ii) diffusion-based synthetic image generation that captures device-specific feature distributions for pre-training to tackle non-IID data, and (iii) a contribution-aware LoRA configuration and bandwidth allocation algorithm for FedFT to ensure that informative devices are effectively utilized while addressing system heterogeneity. Experimental results on the real-world testbed demonstrate that FeDiSyn reduces completion time by over 52.5% and communication cost by over 97.2%, while achieving comparable accuracy to state-of-the-art solutions.

Problem

• federated fine-tuning
• large vision models
• non-IID data
• system heterogeneity
• resource constraints

Innovation

• federated fine-tuning
• synthetic image pre-training
• scaling law
• diffusion-based generation
• contribution-aware LoRA