StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs

2026-06-22

AI Summary

Existing benchmarks for evaluating the statistical reasoning capabilities of large language models (LLMs) suffer from limited scope and homogeneous question formats. To address this, this work proposes StatABench, the first comprehensive evaluation framework that integrates closed-ended knowledge assessments with open-ended, end-to-end modeling tasks. StatABench comprises Stat-Closed—a collection of 404 multi-format questions spanning 18 statistical topics—and Stat-Open—30 open modeling challenges derived from professional competitions. Evaluation leverages the LangChain MCP framework and a multi-agent system, with scoring performed via a validated LLM-as-Judge protocol. Experiments reveal that even GPT-5.1 achieves only 68.6% accuracy on Stat-Closed, while the best open-source model scores 60.6%; on Stat-Open, top-performing agents attain an average score of 61.86, highlighting significant deficiencies in LLMs’ abilities to invoke appropriate tools, make methodological decisions, and execute complete statistical modeling workflows.

Abstract

Statistical analysis is a broad, complex field requiring both domain knowledge and tool proficiency. While prior work has evaluated large language models (LLMs) in this domain, existing benchmarks remain limited in scope and format. To bridge this gap, we introduce StatABench (Statistical AnalysisBenchmark), a benchmark designed to systematically assess LLMs' statistical analysis capabilities. StatABench comprises two complementary components: Stat-Closed, containing 404 questions across 18 statistical topics in multiple formats (multiple-choice, fill-in-the-blank, decision-making, and practical application), and Stat-Open, featuring 30 complex open-ended modeling tasks adapted from professional competitions. We evaluate diverse LLMs using the LangChain MCP framework and multiple data science agents, and assess Stat-Open solutions via a validated LLM-as-Judge protocol. Experiments show that even GPT-5.1 achieves only 68.6% on Stat-Closed, while the best open-source model reaches 60.6%. On Stat-Open, the top agent framework scores 61.86 on average. These results reveal the gap between current LLMs and reliable statistical analysis, highlighting persistent challenges in tool-grounded reasoning, methodological decision-making, and end-to-end statistical modeling.

Problem

• statistical analysis
• large language models
• benchmark
• evaluation
• LLM capabilities

Innovation

• StatABench
• statistical analysis benchmark
• LLM evaluation
• tool-grounded reasoning
• LLM-as-Judge