Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithms

2026-04-22

AI Summary

This work addresses the fundamental challenge of achieving centralized performance in decentralized learning without sharing local data. The authors propose a novel decentralized framework based on Empirical Risk Minimization with Relative Entropy Regularization (ERM-RER), wherein clients communicate only forward and backward messages consisting of locally generated Gibbs measures used as reference distributions. Theoretical analysis establishes, for the first time, that this approach exactly recovers the performance of centralized ERM-RER, provided the regularization strength is appropriately scaled with respect to the local sample size. This result establishes a provably equivalent alternative to centralized training within the federated learning paradigm, eliminating the need for raw data exchange while preserving statistical optimality.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

In this paper, it is shown, for the first time, that centralized performance is achievable in decentralized learning without sharing the local datasets. Specifically, when clients adopt an empirical risk minimization with relative-entropy regularization (ERM-RER) learning framework and a forward-backward communication between clients is established, it suffices to share the locally obtained Gibbs measures to achieve the same performance as that of a centralized ERM-RER with access to all the datasets. The core idea is that the Gibbs measure produced by client~$k$ is used, as reference measure, by client~$k+1$. This effectively establishes a principled way to encode prior information through a reference measure. In particular, achieving centralized performance in the decentralized setting requires a specific scaling of the regularization factors with the local sample sizes. Overall, this result opens the door to novel decentralized learning paradigms that shift the collaboration strategy from sharing data to sharing the local inductive bias via the reference measures over the set of models.

Problem

• Decentralized Machine Learning
• Centralized Performance
• Gibbs Algorithms
• Data Privacy
• Inductive Bias

Innovation

• decentralized learning
• Gibbs algorithm
• relative-entropy regularization
• reference measure
• inductive bias