Soft Separation and Distillation: Toward Global Uniformity in Federated Unsupervised Learning

2025-08-02

AI Summary

Federated unsupervised learning (FUL) struggles to ensure global representation uniformity under non-IID data: while local representations exhibit good uniformity per client, inter-client representation divergence remains substantial after aggregation. To address this, we propose a synergistic framework combining soft separation and projector distillation. Soft separation encourages clients to expand their feature spaces in complementary directions, mitigating representation collapse; projector distillation aligns local optimization objectives with the global one, reducing inconsistency between representation quality and learning goals. Integrating self-supervised learning, federated aggregation, and knowledge distillation, our method jointly optimizes representation uniformity at both local and global levels. Experiments on cross-institutional and cross-device non-IID benchmarks demonstrate significant improvements in representation consistency and downstream task performance, validating the method’s effectiveness and generalizability.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Federated Unsupervised Learning (FUL) aims to learn expressive representations in federated and self-supervised settings. The quality of representations learned in FUL is usually determined by uniformity, a measure of how uniformly representations are distributed in the embedding space. However, existing solutions perform well in achieving intra-client (local) uniformity for local models while failing to achieve inter-client (global) uniformity after aggregation due to non-IID data distributions and the decentralized nature of FUL. To address this issue, we propose Soft Separation and Distillation (SSD), a novel approach that preserves inter-client uniformity by encouraging client representations to spread toward different directions. This design reduces interference during client model aggregation, thereby improving global uniformity while preserving local representation expressiveness. We further enhance this effect by introducing a projector distillation module to address the discrepancy between loss optimization and representation quality. We evaluate SSD in both cross-silo and cross-device federated settings, demonstrating consistent improvements in representation quality and task performance across various training scenarios. Our results highlight the importance of inter-client uniformity in FUL and establish SSD as an effective solution to this challenge. Project page: https://ssd-uniformity.github.io/

Problem

• Achieving global uniformity in federated unsupervised learning
• Addressing non-IID data distribution challenges in FUL
• Improving inter-client uniformity while preserving local expressiveness

Innovation

• Soft Separation and Distillation for global uniformity
• Encourages client representations to spread differently
• Projector distillation enhances loss optimization quality