AugFL: Augmenting Federated Learning with Pretrained Models

2025-03-04

AI Summary

To address the challenge of client data scarcity in federated learning—arising from privacy constraints or limited device storage—this paper proposes a pretraining-empowered regularized federated meta-learning framework. The server leverages a private pretrained model to guide client-specific adaptation without requiring local model uploads or imposing additional computational overhead on clients. We devise an inexact alternating direction method of multipliers (ADMM) optimization algorithm and, for the first time under nonconvex settings, jointly characterize theoretical bounds on knowledge transfer gain, communication complexity, and adaptation performance. By integrating pretrained knowledge distillation with structured regularization, our method significantly accelerates convergence and improves generalization accuracy across clients in low-data regimes. Extensive experiments on multiple benchmark datasets demonstrate consistent superiority over state-of-the-art federated learning approaches.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated Learning (FL) has garnered widespread interest in recent years. However, owing to strict privacy policies or limited storage capacities of training participants such as IoT devices, its effective deployment is often impeded by the scarcity of training data in practical decentralized learning environments. In this paper, we study enhancing FL with the aid of (large) pre-trained models (PMs), that encapsulate wealthy general/domain-agnostic knowledge, to alleviate the data requirement in conducting FL from scratch. Specifically, we consider a networked FL system formed by a central server and distributed clients. First, we formulate the PM-aided personalized FL as a regularization-based federated meta-learning problem, where clients join forces to learn a meta-model with knowledge transferred from a private PM stored at the server. Then, we develop an inexact-ADMM-based algorithm, AugFL, to optimize the problem with no need to expose the PM or incur additional computational costs to local clients. Further, we establish theoretical guarantees for AugFL in terms of communication complexity, adaptation performance, and the benefit of knowledge transfer in general non-convex cases. Extensive experiments corroborate the efficacy and superiority of AugFL over existing baselines.

Problem

• Enhance Federated Learning using pre-trained models
• Reduce data scarcity in decentralized learning environments
• Optimize FL without exposing pre-trained models

Innovation

• Uses pre-trained models to enhance Federated Learning
• Develops AugFL algorithm for efficient optimization
• Ensures privacy and reduces computational costs