A Thorough Assessment of the Non-IID Data Impact in Federated Learning

2025-03-21

AI Summary

Non-independent and identically distributed (non-IID) data severely degrade federated learning (FL) performance, yet systematic characterization—particularly of spatiotemporal skew—remains lacking. Method: This work introduces a unified taxonomy of non-IID heterogeneity encompassing label, feature, quantity, and *spatiotemporal* skew—the first to explicitly incorporate temporal and spatial dimensions—and quantifies each under both controlled and real-world settings. We propose a Hellinger-distance (HD)-based metric to uniformly measure non-IID severity and identify critical HD thresholds triggering sharp model accuracy degradation. Using benchmark evaluations across four state-of-the-art heterogeneity-robust algorithms (e.g., FedProx, SCAFFOLD), we analyze convergence behavior and accuracy trends. Contribution/Results: Label and spatiotemporal skews exert the strongest negative impact on accuracy; moreover, under extreme non-IID conditions, clear critical points emerge where convergence deteriorates and performance collapses. Our findings provide reproducible empirical evidence and quantitative guidelines for designing robust FL systems.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Federated learning (FL) allows collaborative machine learning (ML) model training among decentralized clients' information, ensuring data privacy. The decentralized nature of FL deals with non-independent and identically distributed (non-IID) data. This open problem has notable consequences, such as decreased model performance and more significant convergence times. Despite its importance, experimental studies systematically addressing all types of data heterogeneity (a.k.a. non-IIDness) remain scarce. We aim to fill this gap by assessing and quantifying the non-IID effect through a thorough empirical analysis. We use the Hellinger Distance (HD) to measure differences in distribution among clients. Our study benchmarks four state-of-the-art strategies for handling non-IID data, including label, feature, quantity, and spatiotemporal skewness, under realistic and controlled conditions. This is the first comprehensive analysis of the spatiotemporal skew effect in FL. Our findings highlight the significant impact of label and spatiotemporal skew non-IID types on FL model performance, with notable performance drops occurring at specific HD thresholds. Additionally, the FL performance is heavily affected mainly when the non-IIDness is extreme. Thus, we provide recommendations for FL research to tackle data heterogeneity effectively. Our work represents the most extensive examination of non-IIDness in FL, offering a robust foundation for future research.

Problem

• Assessing impact of non-IID data in federated learning
• Measuring distribution differences using Hellinger Distance
• Benchmarking strategies for handling data heterogeneity

Innovation

• Uses Hellinger Distance to measure data distribution differences
• Benchmarks four strategies for handling non-IID data
• First comprehensive analysis of spatiotemporal skew in FL