Large Language Models for Data Synthesis

2025-05-20

AI Summary

Conventional high-dimensional heterogeneous data synthesis methods rely on strong parametric assumptions, while large language models (LLMs) suffer from statistical misalignment and context-length limitations. Method: We propose LLMSynthor—a framework that transforms LLMs into structure-aware, nonparametric copula simulators. It introduces the novel LLM Proposal Sampling mechanism, enabling distributional feedback-driven iterative optimization to minimize statistical divergence between real and synthetic data in the summary statistics space. The approach integrates nonparametric copula modeling, multimodal heterogeneous adaptation, and differentiable statistical alignment—eliminating the need for manual structural design. Contribution/Results: Evaluated on privacy-sensitive domains—including e-commerce, demographics, and transportation—LLMSynthor generates synthetic data with high statistical fidelity, strong downstream task utility, and cross-dataset generalizability, outperforming existing methods in both quality and practical applicability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Generating synthetic data that faithfully captures the statistical structure of real-world distributions is a fundamental challenge in data modeling. Classical approaches often depend on strong parametric assumptions or manual structural design and struggle in high-dimensional or heterogeneous domains. Recent progress in Large Language Models (LLMs) reveals their potential as flexible, high-dimensional priors over real-world distributions. However, when applied to data synthesis, standard LLM-based sampling is inefficient, constrained by fixed context limits, and fails to ensure statistical alignment. Given this, we introduce LLMSynthor, a general framework for data synthesis that transforms LLMs into structure-aware simulators guided by distributional feedback. LLMSynthor treats the LLM as a nonparametric copula simulator for modeling high-order dependencies and introduces LLM Proposal Sampling to generate grounded proposal distributions that improve sampling efficiency without requiring rejection. By minimizing discrepancies in the summary statistics space, the iterative synthesis loop aligns real and synthetic data while gradually uncovering and refining the latent generative structure. We evaluate LLMSynthor in both controlled and real-world settings using heterogeneous datasets in privacy-sensitive domains (e.g., e-commerce, population, and mobility) that encompass both structured and unstructured formats. The synthetic data produced by LLMSynthor shows high statistical fidelity, practical utility, and cross-data adaptability, positioning it as a valuable tool across economics, social science, urban studies, and beyond.

Problem

• Generating synthetic data with real-world statistical fidelity
• Overcoming inefficiency in standard LLM-based sampling methods
• Ensuring statistical alignment between real and synthetic data

Innovation

• LLMSynthor framework for structure-aware data synthesis
• LLM Proposal Sampling improves efficiency without rejection
• Iterative synthesis aligns real and synthetic statistics