A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent

2024-07-19

arXiv.org

AI Summary

To address convergence failure in distributed learning caused by adversarial gradient corruption, this paper proposes a robust distributed optimization algorithm based on Lazy Mirror Descent. The method integrates mirror descent into the distributed gradient descent framework—its first such application—to enhance resilience against corrupted gradients. It further introduces an adaptive step-size strategy that amortizes the impact of corruption over time, simultaneously ensuring robustness and accelerating convergence. Theoretically, the algorithm is proven to converge for both strongly convex and general convex loss functions under adversarial perturbations. Empirical evaluation on MNIST—including linear regression, SVM, and softmax classification—demonstrates that the proposed approach significantly outperforms standard distributed gradient descent under various malicious node contamination settings, validating its effectiveness and practicality.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Distributed Search

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Distributed gradient descent algorithms have come to the fore in modern machine learning, especially in parallelizing the handling of large datasets that are distributed across several workers. However, scant attention has been paid to analyzing the behavior of distributed gradient descent algorithms in the presence of adversarial corruptions instead of random noise. In this paper, we formulate a novel problem in which adversarial corruptions are present in a distributed learning system. We show how to use ideas from (lazy) mirror descent to design a corruption-tolerant distributed optimization algorithm. Extensive convergence analysis for (strongly) convex loss functions is provided for different choices of the stepsize. We carefully optimize the stepsize schedule to accelerate the convergence of the algorithm, while at the same time amortizing the effect of the corruption over time. Experiments based on linear regression, support vector classification, and softmax classification on the MNIST dataset corroborate our theoretical findings.

Problem

• Address adversarial corruptions in distributed learning systems.
• Develop corruption-tolerant distributed gradient descent algorithm.
• Optimize stepsize for accelerated convergence and corruption amortization.

Innovation

• Mirror Descent-Based Algorithm
• Corruption-Tolerant Optimization
• Stepsize Schedule Optimization