RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis

2025-06-16

AI Summary

Existing tabular evaluation benchmarks are largely confined to simple flat structures or outdated datasets, lacking systematic assessment of real-world complex hierarchical tables—particularly those with nested headers and multimodal formats. To address this gap, we propose TableBench, the first comprehensive benchmark specifically designed for complex hierarchical tables, supporting multimodal inputs (LaTeX, HTML, PNG) and diverse reasoning tasks. We further introduce TreeThinker, a novel tree-structured parsing method that explicitly models header hierarchies to enhance structural awareness. Additionally, we establish a unified evaluation framework for both LLMs and multimodal LLMs (MLLMs). Extensive evaluation across 25 state-of-the-art models demonstrates TableBench’s high difficulty and practical relevance. TreeThinker achieves substantial gains in hierarchical table question answering—improving accuracy by an average of +12.3%—thereby empirically validating the critical role of structured header modeling in tabular reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With the rapid advancement of Large Language Models (LLMs), there is an increasing need for challenging benchmarks to evaluate their capabilities in handling complex tabular data. However, existing benchmarks are either based on outdated data setups or focus solely on simple, flat table structures. In this paper, we introduce RealHiTBench, a comprehensive benchmark designed to evaluate the performance of both LLMs and Multimodal LLMs (MLLMs) across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. RealHiTBench also includes a diverse collection of tables with intricate structures, spanning a wide range of task types. Our experimental results, using 25 state-of-the-art LLMs, demonstrate that RealHiTBench is indeed a challenging benchmark. Moreover, we also develop TreeThinker, a tree-based pipeline that organizes hierarchical headers into a tree structure for enhanced tabular reasoning, validating the importance of improving LLMs' perception of table hierarchies. We hope that our work will inspire further research on tabular data reasoning and the development of more robust models. The code and data are available at https://github.com/cspzyy/RealHiTBench.

Problem

• Evaluating LLMs on complex hierarchical table analysis
• Addressing lack of realistic benchmarks for tabular data
• Improving table hierarchy perception in language models

Innovation

• Introduces RealHiTBench for complex tabular data evaluation
• Develops TreeThinker for hierarchical table reasoning
• Supports multiple input formats like LaTeX, HTML, PNG