IMAGIC-500: IMputation benchmark on A Generative Imaginary Country (500k samples)

2025-06-10

AI Summary

Social-economic data are rarely publicly available due to stringent privacy constraints, hindering reproducible benchmarking for missing-data imputation. To address this, we introduce IMAGIC-500—the first large-scale (500k samples), privacy-compliant, generative synthetic dataset designed specifically for socio-economic research. It supports systematic evaluation under diverse missingness mechanisms (MCAR, MAR, MNAR) and multiple missingness rates. IMAGIC-500 innovatively integrates real-world survey structures—hierarchically modeled from World Bank data—with state-of-the-art synthetic data generation. Within a unified framework, it jointly evaluates imputation accuracy, computational efficiency, and downstream task performance (e.g., educational attainment prediction). We benchmark over 20 methods, including MICE, Random Forest, XGBoost, VAEs, and diffusion models. The dataset and evaluation code are fully open-sourced, establishing the first high-quality, publicly accessible benchmark for missing-data imputation in socio-economic domains.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Humans and AI: Crowd Sourcing and Human Computation
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data generation and simulation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Missing data imputation in tabular datasets remains a pivotal challenge in data science and machine learning, particularly within socioeconomic research. However, real-world socioeconomic datasets are typically subject to strict data protection protocols, which often prohibit public sharing, even for synthetic derivatives. This severely limits the reproducibility and accessibility of benchmark studies in such settings. Further, there are very few publicly available synthetic datasets. Thus, there is limited availability of benchmarks for systematic evaluation of imputation methods on socioeconomic datasets, whether real or synthetic. In this study, we utilize the World Bank's publicly available synthetic dataset, Synthetic Data for an Imaginary Country, which closely mimics a real World Bank household survey while being fully public, enabling broad access for methodological research. With this as a starting point, we derived the IMAGIC-500 dataset: we select a subset of 500k individuals across approximately 100k households with 19 socioeconomic features, designed to reflect the hierarchical structure of real-world household surveys. This paper introduces a comprehensive missing data imputation benchmark on IMAGIC-500 under various missing mechanisms (MCAR, MAR, MNAR) and missingness ratios (10%, 20%, 30%, 40%, 50%). Our evaluation considers the imputation accuracy for continuous and categorical variables, computational efficiency, and impact on downstream predictive tasks, such as estimating educational attainment at the individual level. The results highlight the strengths and weaknesses of statistical, traditional machine learning, and deep learning imputation techniques, including recent diffusion-based methods. The IMAGIC-500 dataset and benchmark aim to facilitate the development of robust imputation algorithms and foster reproducible social science research.

Problem

• Lack of public benchmarks for socioeconomic data imputation
• Need for reproducible missing data imputation evaluation methods
• Limited access to synthetic datasets mimicking real surveys

Innovation

• Uses synthetic dataset mimicking real surveys
• Benchmarks imputation under various missing mechanisms
• Evaluates accuracy, efficiency, and downstream impact