Privacy-Preserving Conformal Prediction Under Local Differential Privacy

2025-05-21

AI Summary

In local differential privacy (LDP) settings where ground-truth labels are inaccessible and the aggregator is untrusted, existing conformal prediction methods fail due to reliance on label-based calibration. Method: This work introduces the first LDP-conformal prediction framework that eliminates the need for true-label calibration. It comprises: (1) a k-ary randomized response mechanism to privatize categorical labels under LDP; and (2) a score-noising strategy based on binary search, enabling robust nonconformity score construction on perturbed data. Contribution/Results: We theoretically establish finite-sample validity—guaranteeing coverage ≥ 1−α with high probability. Experiments demonstrate stable coverage near the target level even under stringent privacy budgets (ε ≤ 1), validating practical utility in high-stakes applications such as medical image classification and large language model querying. The framework supports two deployment paradigms: model-agnostic (client-side inference) and model-embedded (client-side model execution).

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Conformal prediction (CP) provides sets of candidate classes with a guaranteed probability of containing the true class. However, it typically relies on a calibration set with clean labels. We address privacy-sensitive scenarios where the aggregator is untrusted and can only access a perturbed version of the true labels. We propose two complementary approaches under local differential privacy (LDP). In the first approach, users do not access the model but instead provide their input features and a perturbed label using a k-ary randomized response. In the second approach, which enforces stricter privacy constraints, users add noise to their conformity score by binary search response. This method requires access to the classification model but preserves both data and label privacy. Both approaches compute the conformal threshold directly from noisy data without accessing the true labels. We prove finite-sample coverage guarantees and demonstrate robust coverage even under severe randomization. This approach unifies strong local privacy with predictive uncertainty control, making it well-suited for sensitive applications such as medical imaging or large language model queries, regardless of whether users can (or are willing to) compute their own scores.

Problem

• Ensuring privacy in conformal prediction with untrusted aggregators
• Providing accurate class sets under local differential privacy
• Maintaining data and label privacy in sensitive applications

Innovation

• Uses k-ary randomized response for perturbed labels
• Employs binary search response for noisy conformity scores
• Computes conformal threshold directly from noisy data