Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data

2026-07-02

AI Summary

This work addresses the limited robustness and lack of theoretical grounding of distributed self-supervised learning under non-IID data. The authors systematically analyze the behavior of contrastive learning (CL) and masked image modeling (MIM) in heterogeneous data settings, theoretically demonstrating that MIM is inherently more robust than CL and uncovering the critical role of network connectivity in performance. Building on these insights, they propose a novel loss function, termed MAR, which enables effective alignment of local and global features. Extensive experiments across diverse architectures and distributed configurations validate that MAR consistently and significantly improves model performance on non-IID data, outperforming existing approaches.

Abstract

Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theoretical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.

Problem

• distributed self-supervised learning
• non-IID data
• data heterogeneity
• robustness
• theoretical analysis

Innovation

• distributed self-supervised learning
• non-IID robustness
• Masked Image Modeling
• theoretical analysis
• MAR loss