Same Content, Different Representations: A Controlled Study for Table QA

2025-09-26

AI Summary

This work investigates how tabular representation formats affect Table QA performance, addressing the underexplored issue of representation diversity in existing benchmarks. We propose the first controlled-variable experimental framework: preserving identical semantic content across tables while systematically comparing model performance on structured versus semi-structured representations. To enable fine-grained analysis, we construct a diagnostic benchmark stratified by scale, connectivity requirements, query complexity, and schema quality. Paired tables are generated via a text-to-table pipeline, ensuring semantic equivalence. We conduct multi-paradigm evaluation across SQL-based models, large language models (LLMs), and hybrid approaches. Results reveal that SQL-based methods achieve high accuracy on structured tables but suffer from poor generalization; LLMs exhibit strong robustness yet lower precision; hybrid methods significantly outperform others under noisy schemas, with this advantage amplifying as table scale and query complexity increase.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: Question Answering
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Table Question Answering (Table QA) in real-world settings must operate over both structured databases and semi-structured tables containing textual fields. However, existing benchmarks are tied to fixed data formats and have not systematically examined how representation itself affects model performance. We present the first controlled study that isolates the role of table representation by holding content constant while varying structure. Using a verbalization pipeline, we generate paired structured and semi-structured tables, enabling direct comparisons across modeling paradigms. To support detailed analysis, we introduce a diagnostic benchmark with splits along table size, join requirements, query complexity, and schema quality. Our experiments reveal consistent trade-offs: SQL-based methods achieve high accuracy on structured inputs but degrade on semi-structured data, LLMs exhibit flexibility but reduced precision, and hybrid approaches strike a balance, particularly under noisy schemas. These effects intensify with larger tables and more complex queries. Ultimately, no single method excels across all conditions, and we highlight the central role of representation in shaping Table QA performance. Our findings provide actionable insights for model selection and design, paving the way for more robust hybrid approaches suited for diverse real-world data formats.

Problem

• Examining how table representation affects model performance in QA
• Comparing SQL, LLM and hybrid methods across structured/semi-structured data
• Analyzing performance trade-offs under varying table sizes and query complexity

Innovation

• Generating paired structured and semi-structured tables
• Introducing diagnostic benchmark with multiple splits
• Evaluating trade-offs between SQL, LLMs, and hybrid methods