DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis

2026-05-04

AI Summary

This work addresses the limitation of existing benchmarks, which predominantly focus on final-answer accuracy under known data and fail to evaluate agents’ exploratory reasoning in low-prior, real-world settings. To bridge this gap, we introduce DataClaw—a process-oriented evaluation benchmark comprising approximately 2.06 million real-world records with inherent noise and 492 cross-domain tasks derived from think-tank consultations, each annotated with intermediate milestones to enable fine-grained process-level assessment. Employing a novel process-driven evaluation framework and systematic testing with large language models, we find that current mainstream models perform poorly overall (7 out of 8 models achieve below 50% accuracy). Nevertheless, process-level analysis uncovers partially effective reasoning pathways and strategic variations, thereby transcending the constraints of conventional evaluation paradigms that rely solely on final-answer correctness.

Abstract

Evaluating autonomous data analysis agents requires testing their ability to perform exploratory analysis in underexplored data environments. However, many existing benchmarks emphasize final answer accuracy in prior-guided data settings and provide limited support for reasoning process evaluation. We introduce DataClaw, a process-oriented benchmark for exploratory real-world data analysis. DataClaw contains approximately 2.06 million real-world records across enterprise, industry and policy domains, with native data noise preserved. It further includes 492 cross-domain tasks derived from think-tank consulting scenarios, each annotated with intermediate milestones for process-level evaluation. These annotations allow DataClaw to measure how far an agent progresses and where its reasoning breaks down. Experiments with eight advanced LLMs show that current agents remain far from reliable in this setting, with seven models achieving below 50% overall accuracy. Process analysis further reveals partial progress hidden behind wrong answers and distinct exploration strategies across models. Overall, DataClaw provides a less data constrained diagnostic testbed for probing the capability boundaries of autonomous data-analysis agents.

Problem

• autonomous data analysis
• exploratory data analysis
• agent benchmark
• reasoning process evaluation
• real-world data

Innovation

• process-oriented benchmark
• exploratory data analysis
• real-world data
• autonomous agents
• reasoning process evaluation