Label Granularity Skew in Federated Learning with Hierarchical Image Classification

2026-08-10

AI Summary

This work addresses the label granularity skew arising in federated hierarchical image classification due to heterogeneous annotation granularities across clients. It formally characterizes this previously unexplored form of statistical heterogeneity and proposes Federated Branch-Decoupled Fine-Tuning (FedBDFT). FedBDFT constructs client-specific local label hierarchies and integrates WordNet-guided hierarchy coarsening with conditional Softmax classifiers, enabling decoupled fine-tuning and aggregation of branch classifiers within a federated learning framework. Experimental results demonstrate that FedBDFT substantially improves generalization performance on CIFAR-100, TinyImageNet, and ImageNet, achieving average accuracy gains of 27.9% and 56.4% under granularity skew levels of 0.6 and 0.9, respectively, while effectively preserving hierarchical semantic structure even in zero-shot settings.

Abstract

Federated learning enables privacy-preserving collaboration across distributed devices without centralizing local data. However, clients may differ not only in data distributions but also in domain knowledge and annotation capabilities. In this paper, we introduce label granularity skew, a new form of statistical heterogeneity in federated hierarchical classification, in which clients provide taxonomy-consistent labels at different levels of detail within a shared class hierarchy. To model this heterogeneity, we generate client-specific local label hierarchies using a probabilistic relational neighbor classifier and construct a WordNet-guided hierarchy via silhouette score-based coarsening. Our analysis shows that strongly coupled hierarchical models are sensitive to incomplete supervision, while the conditional softmax classifier is more robust. Based on this insight, we propose Branch-wise Decoupled Fine-Tuning (BDFT) and its federated version, FedBDFT, which fine-tune branch-wise classifiers and aggregate them through federated optimization. Experiments on CIFAR-100, TinyImageNet, and ImageNet show that FedBDFT substantially improves robustness under severe label granularity skew, with average gains of 27.9% and 56.4% at skewness levels of 0.6 and 0.9, respectively. Zero-shot results further indicate that FedBDFT better preserves hierarchical representations for unseen fine-grained classes. These findings demonstrate its effectiveness for federated hierarchical classification with heterogeneous label granularities.

Problem

• federated learning
• hierarchical classification
• label granularity skew
• statistical heterogeneity
• class hierarchy

Innovation

• label granularity skew
• federated hierarchical classification
• conditional softmax
• Branch-wise Decoupled Fine-Tuning
• heterogeneous supervision