WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild

2026-05-01

AI Summary

Existing table understanding benchmarks predominantly rely on structured text or idealized images, failing to capture the challenges posed by complex layouts and diverse domains in real-world scenarios. To address this gap, this work proposes WildTableBench—the first question-answering benchmark centered on naturally occurring “in-the-wild” table images. It comprises 402 high-information-density tables and 928 fine-grained questions, organized into five categories and seventeen subtasks. Leveraging human-collected and meticulously annotated data, we conduct a systematic evaluation of 21 state-of-the-art multimodal models. The results reveal that only one model surpasses 50% accuracy, with others ranging from 4.1% to 49.9%, highlighting significant deficiencies in structural perception and numerical reasoning. This benchmark thus fills a critical void in evaluating unstructured table comprehension.

Abstract

Using multimodal foundation models to analyze table images is a high-value yet challenging application in consumer and enterprise scenarios. Despite its importance, current evaluations rely largely on structured-text tables or clean rendered images, leaving the visual complexity of in-the-wild table images underexplored. Such images feature varied layouts and diverse domains that demand sophisticated structural perception and numerical reasoning. To bridge this gap, we introduce WildTableBench, the first question-answering benchmark for naturally occurring table images from real-world settings. WildTableBench comprises 402 high-information-density table images collected from online forums and websites across diverse domains, together with 928 manually annotated and verified questions spanning 17 subtypes across five categories. We evaluate 21 frontier proprietary and open-source multimodal foundation models on this benchmark. Only one model exceeds 50% accuracy, while all remaining models range from 4.1% to 49.9%. We further conduct diagnostic analyses to characterize model failures and reveal persistent weaknesses in structural perception and reasoning. These results and analyses provide useful insights into current model capabilities and establish WildTableBench as a valuable diagnostic benchmark for table image understanding.

Problem

• table understanding
• multimodal foundation models
• in-the-wild images
• structural perception
• numerical reasoning

Innovation

• table understanding
• multimodal foundation models
• in-the-wild benchmark
• structural perception
• numerical reasoning