Statistically Indistinguishable, Operationally Distinct: A Formal Barrier for Tabular Foundation Models

2026-06-27

AI Summary

This work addresses a fundamental limitation of tabular foundation models that rely solely on numerical data: their inability to distinguish between database states that are statistically indistinguishable yet differ in operational legality. The authors propose the Operational Turing Test (OTT), which constructs pairs of states that are statistically identical but vary in compliance with explicit operational rules. Through this framework, they formally demonstrate that such models face an inherent identifiability barrier—not due to insufficient model capacity, but because of the absence of explicit rule knowledge. Empirical results show that purely numerical models plateau at 50% accuracy, equivalent to random guessing, whereas a classifier augmented with seven rule-derived audit features achieves perfect (100%) accuracy. Even when integrated with SQL executors, state-of-the-art large language models fail to reliably discern legally valid states.

Abstract

Tabular foundation models cannot reason about data produced by running systems without access to the rules that govern them. We make this statement falsifiable. The \emph{Operational Turing Test} (OTT) constructs pairs of legal and rule-violating database states whose $1$- and $2$-way column-value marginals match to a total variation of $<0.02$; Le~Cam's lemma then bounds any values-only classifier at $\geq0.49$ Bayes error. Three values-only baselines (XGBoost, TabICL, TabPFN) hit the bound exactly (accuracy $0.50$, pre-registered two one-sided tests (TOST) $p<0.002$), raw row-level access does not help, exposing relational value consistency closes most of the gap, and only a classifier fed by seven executable rule-derived audits reaches $1.00$ classification accuracy. In three matched $100$-state frontier large-language-model (LLM) runs, models given the schema, trigger source, rule tables, and state files classify at most $2/50$ legal states as LEGAL; GPT-5.5 accepts $0/50$ legal states even with higher reasoning effort and a Structured Query Language (SQL) executor. The access-ladder pattern also appears on a second schema with structurally distinct rule families (banking ledger: cross-row balance, cumulative aggregate). The barrier is identifiability, not capacity: scale, data, and richer features cannot cross it without operational grounding.

Problem

• tabular foundation models
• operational grounding
• identifiability
• relational consistency
• rule violation detection

Innovation

• Operational Turing Test
• tabular foundation models
• identifiability barrier
• relational consistency
• rule-based auditing