TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data

2026-07-04

AI Summary

Existing evaluation methods for synthetic tabular data fail to account for the structural fidelity of analytical SQL queries, limiting their ability to reflect real-world data usability. This work proposes TabQueryBench, the first benchmark specifically designed for evaluating query-level fidelity in analytical contexts. It derives 44 reusable SQL templates from 12 public sources and employs strategy-guided template-to-SQL conversion to systematically assess 11 generative models across 49 datasets. The framework enables structure-aware, cross-model and cross-dataset evaluation through multidimensional metrics that distinguish global versus local patterns and head versus tail distributions. Experiments reveal that current models exhibit generally insufficient query fidelity—peaking at only 0.75—with pronounced weaknesses in handling high-cardinality categorical values and tail distributions, while also uncovering a trade-off between fidelity and generation cost.

Abstract

Synthetic tabular data support use cases like data sharing, model development under access restrictions, and rapid prototyping of analytical workflows. Modern generative models are evaluated by their statistical similarity, correlation structure, privacy, and downstream machine-learning utility. However, such evaluations leave a gap: they rarely test the structure that matters for analytical queries. We present TabQueryBench, a query-centric benchmark that uses SQL-shaped analytical queries as structural assessors for synthetic data fidelity. It provides an extensible foundation for query-centric synthetic-data evaluation. From 12 public sources of analytical queries, TabQueryBench taxonomizes recurring cross-domain logic into 44 reusable query templates and grounds them to each dataset via a policy-guided template-to-SQL pipeline. This makes queries schema-aware while preserving comparability across generative models. Across 49 datasets and 11 generative models, it activates 10-12 templates per dataset, producing more than 100 executable SQL queries per dataset. Our systematic experiments show five main patterns. First, current tabular generative models can have good distance-based fidelity, but they still fall short on query-centric fidelity: RealTabFormer achieves the highest query-centric fidelity, but it only reaches 0.75 +/- 0.15 (REAL data score is 1.00). Second, tabular generative models struggle with very high-cardinality discrete support. Third, SOTA generative models preserve good global conditional query-centric fidelity, but fail more on local queries. Fourth, tail fidelity deteriorates as queries move toward the extreme tail; even the best model recovers only about 40.7% of real rare values. Finally, there is a fidelity-cost tradeoff in tabular generation: BayesNet offers the strongest tradeoff, with slightly lower query-centric fidelity but much lower generation cost.

Problem

• synthetic tabular data
• query-centric evaluation
• data fidelity
• analytical queries
• SQL

Innovation

• query-centric evaluation
• synthetic tabular data
• SQL-shaped queries
• fidelity benchmark
• template-to-SQL pipeline