ChatBench: From Static Benchmarks to Human-AI Evaluation

2025-03-22

Annual Meeting of the Association for Computational Linguistics

AI Summary

Existing LLM benchmarks (e.g., MMLU) evaluate only “AI-alone” performance, failing to reflect real-world human-AI collaboration. Method: We introduce ChatBench—the first benchmark for human-AI collaborative question answering—built from authentic interactions where humans and LLMs jointly solve MMLU questions. It comprises an open-source dataset of 396 questions, 144K answers, and 7,336 dialogue turns, categorized into AI-alone, user-alone, and user-AI settings. We further propose a differentiable user simulator to enable scalable, interactive evaluation. Contribution/Results: Our work establishes the first systematic evaluation paradigm for human-AI collaboration. We find no significant correlation between AI-alone accuracy and user-AI performance, revealing that static benchmarks substantially misestimate actual collaborative efficacy. The user simulator improves evaluation correlation by over 20 percentage points. ChatBench provides both a foundational benchmark and a methodological framework to advance LLM research in human-AI co-intelligence.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

With the rapid adoption of LLM-based chatbots, there is a pressing need to evaluate what humans and LLMs can achieve together. However, standard benchmarks, such as MMLU, measure LLM capabilities in isolation (i.e.,"AI-alone"). Here, we design and conduct a user study to convert MMLU questions into user-AI conversations, by seeding the user with the question and having them carry out a conversation with the LLM to answer their question. We release ChatBench, a new dataset with AI-alone, user-alone, and user-AI data for 396 questions and two LLMs, including 144K answers and 7,336 user-AI conversations. We find that AI-alone accuracy fails to predict user-AI accuracy, with significant differences across multiple subjects (math, physics, and moral reasoning), and we analyze the user-AI conversations to provide insight into how they diverge from AI-alone benchmarks. Finally, we show that fine-tuning a user simulator on a subset of ChatBench improves its ability to estimate user-AI accuracies, increasing correlation on held-out questions by more than 20 points, creating possibilities for scaling interactive evaluation.

Problem

• Evaluating human-AI collaboration in answering questions
• Comparing AI-alone and user-AI performance differences
• Improving user simulator accuracy for interactive evaluation

Innovation

• Converts MMLU questions into user-AI conversations
• Introduces ChatBench dataset for interactive evaluation
• Fine-tunes user simulator to estimate user-AI accuracy