Communication Bounds for the Distributed Experts Problem

2025-01-06

AI Summary

This work addresses communication-efficient collaborative learning among multiple servers in the distributed experts problem under adaptive strong adversaries. The goal is to minimize inter-server communication overhead while achieving near-optimal regret. We propose the first communication-efficient distributed online learning protocol, supporting both message-passing and broadcast communication models, and compatible with diverse aggregation functions—including summation and ℓₚ-norms. We establish a near-optimal regret bound and derive the first conditional communication lower bound, demonstrating that our protocol’s communication complexity is nearly tight. Our analysis integrates distributed online learning theory, communication complexity, and adaptive adversarial modeling to provide rigorous theoretical guarantees. Empirical evaluation on the HPO-B benchmark shows substantial communication savings—up to 92%—while maintaining competitive performance. The work thus bridges deep theoretical insights with practical applicability in resource-constrained distributed learning settings.

Technology Category

• Machine Learning: Online Learning & Bandits
• Game Theory and Economic Paradigms: Adversarial Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

In this work, we study the experts problem in the distributed setting where an expert's cost needs to be aggregated across multiple servers. Our study considers various communication models such as the message-passing model and the broadcast model, along with multiple aggregation functions, such as summing and taking the $ell_p$ norm of an expert's cost across servers. We propose the first communication-efficient protocols that achieve near-optimal regret in these settings, even against a strong adversary who can choose the inputs adaptively. Additionally, we give a conditional lower bound showing that the communication of our protocols is nearly optimal. Finally, we implement our protocols and demonstrate empirical savings on the HPO-B benchmarks.

Problem

• Distributed Learning
• Resource Allocation
• Communication Efficiency

Innovation

• Distributed Expert Problem
• Optimal Strategy
• Resource Efficiency