A Consensus Privacy Metrics Framework for Synthetic Data

2025-03-06

AI Summary

Current privacy assessments of synthetic data lack standardized, quantifiable metrics—particularly for identity, membership, and attribute disclosure risks. Method: We introduce the first expert-consensus–driven privacy measurement framework, employing the Delphi method to systematically define the conceptual boundaries of these three risk categories; we empirically expose the fundamental inadequacy of prevailing similarity-based metrics in privacy evaluation and formally refute the interpretability of non-negligible differential privacy budgets. Contribution/Results: The framework yields an actionable, taxonomy-aware privacy metric recommendation checklist, explicitly identifying critical research gaps. It establishes a rigorous, reproducible, and fine-grained theoretical foundation and practical toolkit for regulatory-compliant privacy assessment of synthetic data.

Technology Category

• Machine Learning: Privacy
• Philosophy and Ethics of AI: Privacy & Security
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Synthetic data generation is one approach for sharing individual-level data. However, to meet legislative requirements, it is necessary to demonstrate that the individuals' privacy is adequately protected. There is no consolidated standard for measuring privacy in synthetic data. Through an expert panel and consensus process, we developed a framework for evaluating privacy in synthetic data. Our findings indicate that current similarity metrics fail to measure identity disclosure, and their use is discouraged. For differentially private synthetic data, a privacy budget other than close to zero was not considered interpretable. There was consensus on the importance of membership and attribute disclosure, both of which involve inferring personal information about an individual without necessarily revealing their identity. The resultant framework provides precise recommendations for metrics that address these types of disclosures effectively. Our findings further present specific opportunities for future research that can help with widespread adoption of synthetic data.

Problem

• Lack of standardized privacy metrics for synthetic data.
• Current metrics fail to measure identity disclosure effectively.
• Need for precise metrics addressing membership and attribute disclosure.

Innovation

• Developed consensus framework for privacy metrics
• Identified limitations in current similarity metrics
• Focused on membership and attribute disclosure risks