A deep cut into Split Federated Self-supervised Learning

2024-06-12

ECML/PKDD

AI Summary

To address privacy leakage and excessive communication overhead caused by shallow model partitioning in distributed self-supervised learning, this paper proposes a depth-adaptive hierarchical model partitioning framework. We first quantitatively analyze the trade-off between partitioning depth and both privacy preservation and communication efficiency. To mitigate client-side model collapse, we design a momentum-based encoder alignment mechanism. Furthermore, we integrate Split Federated Learning with a MoCo-style contrastive learning objective to enforce local feature-space consistency. Our approach achieves state-of-the-art (SOTA) accuracy while reducing total communication volume by up to 42%, significantly enhancing practical deployability. This work breaks away from conventional fixed shallow-partitioning paradigms and establishes a novel framework for jointly optimizing privacy protection and communication efficiency in distributed self-supervised learning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Distributed Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Privacy-enhancing technologies

Abstract

Collaborative self-supervised learning has recently become feasible in highly distributed environments by dividing the network layers between client devices and a central server. However, state-of-the-art methods, such as MocoSFL, are optimized for network division at the initial layers, which decreases the protection of the client data and increases communication overhead. In this paper, we demonstrate that splitting depth is crucial for maintaining privacy and communication efficiency in distributed training. We also show that MocoSFL suffers from a catastrophic quality deterioration for the minimal communication overhead. As a remedy, we introduce Momentum-Aligned contrastive Split Federated Learning (MonAcoSFL), which aligns online and momentum client models during training procedure. Consequently, we achieve state-of-the-art accuracy while significantly reducing the communication overhead, making MonAcoSFL more practical in real-world scenarios.

Problem

• Optimizing network layer division for privacy and efficiency
• Addressing catastrophic quality deterioration in MocoSFL
• Introducing MonAcoSFL to reduce communication overhead

Innovation

• Introduces Momentum-Aligned Split Federated Learning
• Optimizes network layer splitting for privacy
• Reduces communication overhead in distributed training