HARMONY: Bridging the Personalization-Generalization Gap by Mitigating Representation Skew in Heterogeneous Split Federated Learning

2026-05-07

AI Summary

This work addresses the challenge in heterogeneous split federated learning where divergent client model architectures induce feature representation skew, degrading the server’s generalization capability on out-of-distribution (OOD) samples and intensifying the tension between personalization and universality. To resolve this, the authors propose HARMONY, a novel framework that enables hybrid split federated learning with heterogeneous client architectures for the first time. HARMONY employs meta-learning to simulate diverse feature extractors and introduces contrastive learning at the server to align feature representations across clients—without requiring raw label sharing—thereby jointly optimizing personalization and generalization. Extensive experiments demonstrate that HARMONY consistently outperforms existing methods across multiple datasets and model families, achieving up to 43.0% higher test accuracy in non-OOD settings and 28.3% improvement under OOD conditions, while maintaining reasonable inference latency.

Abstract

Mobile devices face diverse resource constraints and non-IID data class distributions, requiring fast on-device inference for local in-distribution (ID) classes and on-demand remote support for client-specific out-of-distribution (OOD) classes. Hybrid split federated learning (Hybrid SFL) couples personalized client-side front ends (supporting early exit) with a generalized server-side backend for fallback inference, balancing accuracy and cost. However, under client architectural heterogeneity, the existing hybrid SFL suffers from representation skew, where features from customized extractors fail to align in the shared space, leading to a sharp degradation in the server model responsible for OOD prediction. We propose HARMONY, the first hybrid SFL framework to support heterogeneous client architectures. HARMONY modifies meta-learning to simulate diverse extractors across parameters and architectures, and to learn to personalize. To mitigate representation skew, HARMONY conducts server-side contrastive learning to align extracted features, neither sacrificing clients' personalization nor sharing raw labels. Compared to the state of the art across multiple datasets and model families, HARMONY improves test accuracy by up to 43.0%/28.3% without/with OOD, respectively, while maintaining acceptable latency.

Problem

• representation skew
• heterogeneous split federated learning
• out-of-distribution prediction
• client architectural heterogeneity
• feature alignment

Innovation

• heterogeneous split federated learning
• representation skew
• meta-learning
• contrastive learning
• out-of-distribution detection