EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models

2026-08-08

AI Summary

This work addresses the challenge of deploying large models in federated learning due to their excessive resource demands on edge clients. To this end, the authors propose a multi-domain federated learning framework that leverages lightweight client-side proxy models trained collaboratively with a server-side base model. A novel bidirectional cross-distillation strategy aligns the feature spaces of the two models, enabling efficient learning of domain-specific LoRA adapters without sharing any private client data. Experimental results demonstrate that the proposed method substantially reduces computational overhead on clients across multiple real-world datasets and low-power devices, while achieving superior or competitive performance compared to state-of-the-art approaches in most domains.

Abstract

Recent data protection laws have accelerated the adoption of Federated Learning (FL) for privacy-preserving decentralized training. Nevertheless, increasing model sizes impose substantial computational demands on client devices, limiting FL applicability in resource-constrained settings. We introduce a novel multi-domain federated learning framework in which lightweight client-side proxy models collaborate with a server-side Foundation Model (FM) to learn new concepts without sharing private data. Our approach, EFFEKT, enables efficient server-side training of domain-specific LoRA adapters while preserving feature-space alignment between the FM and proxy extractors via novel bi-directional cross-distillation strategies. Experiments on multiple real-world datasets and deployments on low-power edge devices demonstrate improvements over state-of-the-art baselines in most considered domains while maintaining lightweight computation at the client side.

Problem

• Federated Learning
• Foundation Models
• Resource-constrained Devices
• Privacy-preserving Learning
• Knowledge Transfer

Innovation

• Federated Learning
• Foundation Models
• LoRA Adapters
• Cross-Distillation
• Edge Devices