On the effects of similarity metrics in decentralized deep learning under distributional shift

2024-09-16

arXiv.org

AI Summary

In decentralized deep learning under distribution shift, collaborator selection remains challenging when no data exchange is permitted. Method: This paper conducts the first systematic empirical evaluation of three similarity metrics—gradient-, feature-, and output-distribution-based (e.g., cosine similarity, MMD, KL divergence)—for model aggregation under non-IID data, within a Decentralized SGD framework and across heterogeneous datasets. Experiments span CIFAR-10/100, Tiny-ImageNet, and medical imaging benchmarks. Contribution/Results: Feature-space similarity demonstrates superior robustness, yielding average accuracy gains of 3.2–7.8% and significantly reducing collaboration failure rates. Based on these findings, we propose an interpretable, reusable metric selection guideline that provides both theoretical grounding and practical guidance for partner identification in decentralized learning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Decentralized Learning (DL) enables privacy-preserving collaboration among organizations or users to enhance the performance of local deep learning models. However, model aggregation becomes challenging when client data is heterogeneous, and identifying compatible collaborators without direct data exchange remains a pressing issue. In this paper, we investigate the effectiveness of various similarity metrics in DL for identifying peers for model merging, conducting an empirical analysis across multiple datasets with distribution shifts. Our research provides insights into the performance of these metrics, examining their role in facilitating effective collaboration. By exploring the strengths and limitations of these metrics, we contribute to the development of robust DL methods.

Problem

• Evaluating similarity metrics in decentralized learning
• Addressing model aggregation with heterogeneous data
• Identifying collaborators without direct data exchange

Innovation

• Decentralized Learning collaboration enhancement
• Similarity metrics for model merging
• Empirical analysis on distribution shifts