Diversity-Driven Learning: Tackling Spurious Correlations and Data Heterogeneity in Federated Models

2025-04-15

AI Summary

To address poor generalization and slow convergence in federated learning (FL) caused by non-independent and identically distributed (Non-IID) client data, class/attribute imbalance, and spurious correlations, this paper proposes FedDiverse—a dynamic client selection algorithm. We further construct the first suite of seven vision benchmark datasets explicitly designed to capture multi-granularity imbalances and spurious correlations. Methodologically, we introduce, for the first time, a systematic six-dimensional metric for quantifying data heterogeneity and establish a novel client selection paradigm grounded in complementary distribution collaboration. Extensive experiments across the seven benchmarks demonstrate that FedDiverse consistently improves the average accuracy of mainstream FL methods by 3.2%, accelerates convergence by 21%, reduces communication and computational overhead, and enhances model robustness against distributional shifts and spurious patterns.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Federated Learning (FL) enables decentralized training of machine learning models on distributed data while preserving privacy. However, in real-world FL settings, client data is often non-identically distributed and imbalanced, resulting in statistical data heterogeneity which impacts the generalization capabilities of the server's model across clients, slows convergence and reduces performance. In this paper, we address this challenge by first proposing a characterization of statistical data heterogeneity by means of 6 metrics of global and client attribute imbalance, class imbalance, and spurious correlations. Next, we create and share 7 computer vision datasets for binary and multiclass image classification tasks in Federated Learning that cover a broad range of statistical data heterogeneity and hence simulate real-world situations. Finally, we propose FedDiverse, a novel client selection algorithm in FL which is designed to manage and leverage data heterogeneity across clients by promoting collaboration between clients with complementary data distributions. Experiments on the seven proposed FL datasets demonstrate FedDiverse's effectiveness in enhancing the performance and robustness of a variety of FL methods while having low communication and computational overhead.

Problem

• Addressing statistical data heterogeneity in Federated Learning
• Mitigating spurious correlations and class imbalance in FL
• Improving model generalization and convergence in decentralized training

Innovation

• Proposes six metrics for statistical data heterogeneity
• Introduces seven diverse FL datasets for real-world simulation
• Develops FedDiverse algorithm for client selection optimization