Decentralized Gradient Descent: Bottleneck Regimes and Budget Complexity

2026-07-13

AI Summary

This work addresses the lack of clear characterization of communication and computation resource requirements for decentralized gradient descent (DGD) to achieve a target accuracy. It proposes a resource-aware analytical framework that introduces two key quantities—the Diversity-to-Network-connectivity Ratio (DNR) and the Gradient-to-Communication-noise Ratio (GCR)—to uncover the dominant bottleneck mechanisms and resource allocation patterns across different error regimes in the optimization process. By integrating multi-stage analysis, network optimization, stochastic gradient theory, and noise modeling, the study derives optimal step-size selection rules and explicit complexity bounds on communication-computation budgets. This is the first work to quantitatively characterize the fundamental trade-offs among data heterogeneity, network connectivity, gradient noise, and communication noise, thereby providing a theoretical foundation for budget-optimal DGD execution strategies.

Abstract

Decentralized gradient descent (DGD) is widely used for solving distributed optimization problems over networks of agents. While its convergence properties are well understood, less is known about the communication and computation resources required to attain a prescribed accuracy. In this paper, we study DGD from a resource-aware perspective and characterize the communication-computation budget required to attain a target error level. We develop a bottleneck-centric framework in which different factors dominate the optimization dynamics at different error scales. Specifically, we identify operating regimes governed by initialization, objective heterogeneity and network connectivity, gradient noise, and communication noise. To capture these effects, we introduce two fundamental quantities: the gradient-Diversity-to-Network-connectivity Ratio (DNR) and the Gradient-to-Communication-noise Ratio (GCR). We show that these quantities determine the sequence of bottlenecks encountered during optimization and the corresponding budget-optimal operating strategy. Using a multi-stage analysis, we derive optimal stepsize selections and explicit budget-complexity bounds that quantify the budget resources required to attain a prescribed accuracy. The resulting expressions reveal how the overall budget decomposes into contributions associated with successive bottlenecks and provide insight into the fundamental tradeoffs among objective heterogeneity, network connectivity, gradient noise, and communication noise.

Problem

• Decentralized Gradient Descent
• Budget Complexity
• Communication-Computation Tradeoff
• Optimization Bottlenecks
• Resource-Aware Optimization

Innovation

• Decentralized Gradient Descent
• Bottleneck Regimes
• Budget Complexity
• Gradient Diversity
• Communication-Computation Tradeoff