Federated Computation of ROC and PR Curves

2025-10-06

AI Summary

In federated learning, servers cannot access raw prediction scores or labels due to data decentralization, privacy constraints, and communication limitations, hindering ROC/PR curve construction. To address this, we propose the first privacy-preserving distributed ROC/PR curve approximation method. Our approach leverages differentially private quantile estimation across clients to reconstruct the global score distribution with minimal communication overhead. We derive tight theoretical upper bounds on AUC and PR-AUC estimation error, enabling optimal trade-offs among accuracy, privacy, and communication cost. Extensive experiments on multiple real-world datasets demonstrate that, under strong privacy guarantees (ε ≤ 2), our method achieves AUC error < 0.01 while reducing total communication volume by over 90%—significantly outperforming existing baselines.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves are fundamental tools for evaluating machine learning classifiers, offering detailed insights into the trade-offs between true positive rate vs. false positive rate (ROC) or precision vs. recall (PR). However, in Federated Learning (FL) scenarios, where data is distributed across multiple clients, computing these curves is challenging due to privacy and communication constraints. Specifically, the server cannot access raw prediction scores and class labels, which are used to compute the ROC and PR curves in a centralized setting. In this paper, we propose a novel method for approximating ROC and PR curves in a federated setting by estimating quantiles of the prediction score distribution under distributed differential privacy. We provide theoretical bounds on the Area Error (AE) between the true and estimated curves, demonstrating the trade-offs between approximation accuracy, privacy, and communication cost. Empirical results on real-world datasets demonstrate that our method achieves high approximation accuracy with minimal communication and strong privacy guarantees, making it practical for privacy-preserving model evaluation in federated systems.

Problem

• Computing ROC and PR curves under federated learning constraints
• Addressing privacy and communication limitations in distributed data
• Approximating evaluation metrics with differential privacy guarantees

Innovation

• Estimates prediction score quantiles with differential privacy
• Provides theoretical bounds on area error trade-offs
• Achieves high accuracy with minimal communication cost