Preserving correlations: A statistical method for generating synthetic data

2024-03-03

arXiv.org

AI Summary

This paper addresses the challenge of simultaneously achieving statistical fidelity and controllable privacy in synthetic data generation. We propose a statistical mapping method based on empirical conditional distributions, which avoids assumptions about underlying data distributions and systematically models conditional dependencies among features to construct an invertible statistical mapping. Privacy is explicitly controlled via tunable parameters that govern information leakage. Our key contribution is the first systematic application of empirical conditional distributions to correlation-preserving synthetic data generation, significantly improving fidelity of higher-order statistics—particularly Pearson correlation matrices. Experiments on synthetic benchmarks, artificial examples, and real-world household energy consumption data from Madeira Island demonstrate that generated data closely preserve the original correlation structure (mean absolute correlation error < 0.05) while providing a well-defined privacy–utility trade-off mechanism.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Generation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We propose a method to generate statistically representative synthetic data from a given dataset. The main goal of our method is for the created data set to mimic the between feature correlations present in the original data, while also offering a tunable parameter to influence the privacy level. In particular, our method constructs a statistical map by using the empirical conditional distributions between the features of the original dataset. We describe in detail our algorithms used both in the construction of a statistical map and how to use this map to generate synthetic observations. This approach is tested in three different ways: with a hand calculated example; a manufactured dataset; and a real world energy-related dataset of consumption/production of households in Madeira Island. We test our method's performance by comparing the datasets using the on Pearson correlation matrix. The proposed methodology is general in the sense that it does not rely on the used test dataset. We expect it to be applicable in a much broader context than indicated here.

Problem

• Generate synthetic data mimicking original feature correlations
• Offer tunable privacy via conditional distribution depth control
• Evaluate method using Pearson correlation matrix comparisons

Innovation

• Generates synthetic data with tunable correlation retention
• Uses empirical conditional distributions for statistical mapping
• Adjusts privacy via correlation depth and resolution limits