Hypernetworks for Model-Heterogeneous Personalized Federated Learning

2025-07-29

AI Summary

In model-heterogeneous personalized federated learning, existing approaches—relying on external data, model decoupling, or partial training—suffer from limited practicality and scalability. To address this, we propose a multi-head hypernetwork-based personalized federated learning framework. The server employs client-specific embedding vectors to drive a multi-head hypernetwork that generates fully customized model parameters for heterogeneous clients, eliminating the need for shared architectures, external data, or local model modifications. Optionally, a lightweight global model can be integrated to improve generalization. Extensive experiments across multiple benchmark datasets and diverse model heterogeneity configurations demonstrate significant gains in personalization performance, achieving state-of-the-art accuracy. The framework ensures strong privacy preservation, architecture-agnosticism, and excellent scalability, establishing a robust and practical new baseline for model-heterogeneous federated learning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Humans and AI: Human-in-the-loop Machine Learning
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advances in personalized federated learning have focused on addressing client model heterogeneity. However, most existing methods still require external data, rely on model decoupling, or adopt partial learning strategies, which can limit their practicality and scalability. In this paper, we revisit hypernetwork-based methods and leverage their strong generalization capabilities to design a simple yet effective framework for heterogeneous personalized federated learning. Specifically, we propose MH-pFedHN, which leverages a server-side hypernetwork that takes client-specific embedding vectors as input and outputs personalized parameters tailored to each client's heterogeneous model. To promote knowledge sharing and reduce computation, we introduce a multi-head structure within the hypernetwork, allowing clients with similar model sizes to share heads. Furthermore, we further propose MH-pFedHNGD, which integrates an optional lightweight global model to improve generalization. Our framework does not rely on external datasets and does not require disclosure of client model architectures, thereby offering enhanced privacy and flexibility. Extensive experiments on multiple benchmarks and model settings demonstrate that our approach achieves competitive accuracy, strong generalization, and serves as a robust baseline for future research in model-heterogeneous personalized federated learning.

Problem

• Addressing client model heterogeneity in federated learning
• Eliminating reliance on external data or model decoupling
• Enhancing privacy and flexibility without disclosing model architectures

Innovation

• Hypernetwork generates personalized client parameters
• Multi-head structure enables shared computation
• Optional global model enhances generalization