Robust Model Evaluation over Large-scale Federated Networks

2024-10-26

arXiv.org

AI Summary

This work addresses the problem of certifying model generalization performance on an unknown target network in federated learning, where the target meta-distribution deviates from the source within a bounded Wasserstein distance or *f*-divergence. We propose a method that provides non-asymptotic, verifiable generalization risk guarantees using only loss queries over clients’ private local data—without uploading raw data. Our approach introduces the first adversarially robust variants of the Glivenko–Cantelli theorem and the Dvoretzky–Kiefer–Wolfowitz inequality, yielding uniform bounds on the cumulative distribution function of risk under meta-distribution shift. By integrating distributionally robust optimization with empirical process theory, we design a privacy-preserving federated query mechanism whose certification relies on polynomial-time computable loss statistics. Theoretically, the estimation error converges to zero at rate jointly dependent on the number of clients *K* and the minimal local sample size. Experiments demonstrate tightness of the bounds and strong cross-domain robustness, significantly outperforming conventional generalization bounds.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Adversarial Attacks & Robustness
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

In this paper, we address the challenge of certifying the performance of a machine learning model on an unseen target network, using measurements from an available source network. We focus on a scenario where heterogeneous datasets are distributed across a source network of clients, all connected to a central server. Specifically, consider a source network"A"composed of $K$ clients, each holding private data from unique and heterogeneous distributions, which are assumed to be independent samples from a broader meta-distribution $mu$. Our goal is to provide certified guarantees for the model's performance on a different, unseen target network"B,"governed by another meta-distribution $mu'$, assuming the deviation between $mu$ and $mu'$ is bounded by either the Wasserstein distance or an $f$-divergence. We derive theoretical guarantees for the model's empirical average loss and provide uniform bounds on the risk CDF, where the latter correspond to novel and adversarially robust versions of the Glivenko-Cantelli theorem and the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality. Our bounds are computable in polynomial time with a polynomial number of queries to the $K$ clients, preserving client privacy by querying only the model's (potentially adversarial) loss on private data. We also establish non-asymptotic generalization bounds that consistently converge to zero as both $K$ and the minimum client sample size grow. Extensive empirical evaluations validate the robustness and practicality of our bounds across real-world tasks.

Problem

• Certify federated learning model performance on unseen networks
• Provide worst-case guarantees for model loss and risk CDF
• Ensure privacy-preserving and efficient performance certification

Innovation

• Certifies model performance under distribution shifts
• Uses Wasserstein distance and f-divergence bounds
• Provides efficient, privacy-preserving, optimal guarantees