Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

2026-07-07

AI Summary

This work addresses the limitations of existing large language model (LLM) evaluation benchmarks, which fail to capture the complexities of real-world data analysis scenarios—particularly multi-table reasoning, integration of external knowledge, and exploratory insight generation. To bridge this gap, the authors propose DataGovBench, a novel benchmark built on open government data that introduces two tasks: Table QA and Table Insight. For the first time, it incorporates realistic multi-table structured environments and supports both agent-based and non-agent frameworks to evaluate LLMs’ capabilities in complex question decomposition, answer generation, visualization, and exploratory analysis. Experimental results reveal that state-of-the-art LLMs perform substantially worse on this benchmark, highlighting their current inadequacies in authentic data analysis tasks and establishing a high-fidelity, challenging standard for future research.

Abstract

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.

Problem

• Large Language Models
• Data Analysis
• Real-World Benchmarks
• Exploratory Data Analysis
• Multi-tabular Datasets

Innovation

• DataGovBench
• Large Language Models
• Exploratory Data Analysis
• Multi-tabular Data
• Real-world Benchmarking