Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality

2026-05-18

AI Summary

Existing synthetic pretraining data are overly idealized and fail to capture the irregularities and failure modes inherent in real-world tabular data, leading to insufficient model robustness. This work proposes O'Prior, a composite realism prior that establishes synthetic prior construction as a critical factor in tabular foundation model quality. O'Prior integrates a hierarchical structural causal model (SCM), heterogeneous marginal distributions, diverse missingness mechanisms, confounding injection, and support-query mismatch into a composable, mechanism-rich generative framework with explicit awareness of distributional shifts. A curriculum control protocol is introduced to prevent information leakage during training. Under fixed architecture and computational budgets, O'Prior substantially improves downstream accuracy and robustness, particularly in scenarios with distributional irregularities. Ablation studies confirm that each component contributes uniquely and indispensably to overall performance.

Abstract

What determines the quality of a tabular foundation model? Unlike language or vision, tabular foundation models acquire their inductive biases almost entirely from synthetic pretraining distributions, yet the design of these distributions remains poorly understood. Standard synthetic priors are too well-behaved: they omit the irregularities and failure modes that determine deployment robustness. We introduce O'Prior, a compositional realism prior built around four coupled components: a hierarchical SCM meta-generator spanning diverse functional families; a modular realism engine covering heterogeneous marginals, missingness, and target transforms; an explicit stress module injecting confounding and support-query mismatch; and a curriculum-governed, leakage-safe generation protocol. To isolate prior design as the scientific variable, we hold architecture, optimizer, and compute budget fixed and vary only the synthetic task distribution. O'Prior yields consistent and substantial improvements in downstream accuracy and robustness across real tabular benchmarks, with gains concentrated in regimes characterized by distributional irregularities. Ablations confirm that mechanism diversity, realism composition, and shift-aware stress each contribute independently, their effects are not interchangeable. These results establish synthetic prior construction as a first-order and largely overlooked determinant of tabular foundation model quality

Problem

• tabular foundation model
• synthetic prior
• inductive bias
• distributional irregularities
• pretraining distribution

Innovation

• synthetic prior
• tabular foundation model
• compositional realism
• distributional robustness
• structural causal model