Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Models

2025-08-19

AI Summary

To address challenges including resource-constrained heterogeneous clients, high communication overhead across multiple rounds, absence of ground-truth labels on public data, and unreliable predictions induced by local data distribution skew, this paper proposes FedOL—the first one-round communication framework for heterogeneous federated knowledge aggregation. Methodologically, FedOL (1) exchanges only soft predictions—rather than model parameters or gradients—on unlabeled public data, enabling architecture-agnostic model ensemble via customized knowledge distillation; (2) introduces an iterative pseudo-labeling mechanism to mitigate local distribution bias; and (3) constructs, server-side and in a fully unsupervised manner, a larger, more robust unified model. Extensive experiments demonstrate that FedOL achieves superior trade-offs among communication efficiency, computational load, and model accuracy—particularly beneficial for resource-limited edge and mobile network environments.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large models, renowned for superior performance, outperform smaller ones even without billion-parameter scales. While mobile network servers have ample computational resources to support larger models than client devices, privacy constraints prevent clients from directly sharing their raw data. Federated Learning (FL) enables decentralized clients to collaboratively train a shared model by exchanging model parameters instead of transmitting raw data. Yet, it requires a uniform model architecture and multiple communication rounds, which neglect resource heterogeneity, impose heavy computational demands on clients, and increase communication overhead. To address these challenges, we propose FedOL, to construct a larger and more comprehensive server model in one-shot settings (i.e., in a single communication round). Instead of model parameter sharing, FedOL employs knowledge distillation, where clients only exchange model prediction outputs on an unlabeled public dataset. This reduces communication overhead by transmitting compact predictions instead of full model weights and enables model customization by allowing heterogeneous model architectures. A key challenge in this setting is that client predictions may be biased due to skewed local data distributions, and the lack of ground-truth labels in the public dataset further complicates reliable learning. To mitigate these issues, FedOL introduces a specialized objective function that iteratively refines pseudo-labels and the server model, improving learning reliability. To complement this, FedOL incorporates a tailored pseudo-label generation and knowledge distillation strategy that effectively integrates diverse knowledge. Simulation results show that FedOL significantly outperforms existing baselines, offering a cost-effective solution for mobile networks where clients possess valuable private data but limited computational resources.

Problem

• Builds larger server model via one-shot federated learning
• Addresses data privacy and client resource heterogeneity constraints
• Mitigates prediction bias from non-IID data without labels

Innovation

• One-shot knowledge distillation using prediction outputs exchange
• Specialized objective function refining pseudo-labels iteratively
• Heterogeneous model architectures with reduced communication overhead