Heterogeneous Federated Learning with Prototype Alignment and Upscaling

2025-07-06

AI Summary

In federated learning, data and model heterogeneity impede sufficient prototype separation, limiting the discriminative capability of prototype-based methods. To address this, we propose ProtoNorm—a server-side framework that jointly enforces prototype alignment and amplification. First, class prototypes are constrained to the unit hypersphere and optimized via a Thomson-problem-inspired formulation to promote globally uniform angular distribution, thereby enhancing inter-class separability. Second, magnitude scaling in Euclidean space is applied to further improve feature discriminability. ProtoNorm integrates prototype normalization, spherical optimization, and feature scaling without incurring additional communication overhead. Extensive experiments on multiple heterogeneous benchmark datasets demonstrate that ProtoNorm consistently outperforms state-of-the-art methods, achieving average accuracy gains of 2.1–4.7% and up to 38.5% improvement in prototype separation (e.g., inter-class cosine distance), validating its effectiveness and generalizability under resource-constrained settings.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Heterogeneity in data distributions and model architectures remains a significant challenge in federated learning (FL). Various heterogeneous FL (HtFL) approaches have recently been proposed to address this challenge. Among them, prototype-based FL (PBFL) has emerged as a practical framework that only shares per-class mean activations from the penultimate layer. However, PBFL approaches often suffer from suboptimal prototype separation, limiting their discriminative power. We propose Prototype Normalization (ProtoNorm), a novel PBFL framework that addresses this limitation through two key components: Prototype Alignment (PA) and Prototype Upscaling (PU). The PA method draws inspiration from the Thomson problem in classical physics, optimizing global prototype configurations on a unit sphere to maximize angular separation; subsequently, the PU method increases prototype magnitudes to enhance separation in Euclidean space. Extensive evaluations on benchmark datasets show that our approach better separates prototypes and thus consistently outperforms existing HtFL approaches. Notably, since ProtoNorm inherits the communication efficiency of PBFL and the PA is performed server-side, it is particularly suitable for resource-constrained environments.

Problem

• Addresses suboptimal prototype separation in federated learning
• Improves discriminative power via prototype alignment and upscaling
• Enhances communication efficiency for resource-constrained environments

Innovation

• Prototype Normalization enhances prototype separation
• Prototype Alignment optimizes global configurations
• Prototype Upscaling boosts Euclidean separation