BIG-Bench Extra Hard

2025-02-26

AI Summary

Existing LLM reasoning benchmarks (e.g., BIG-Bench Hard) suffer from rapid saturation due to accelerating model improvements, rendering them insufficient for discriminating advanced reasoning capabilities. Method: We introduce BBEH—the first systematically harder benchmark for general-purpose reasoning—designed to overcome saturation by re-engineering tasks across core dimensions (mathematical, logical, causal reasoning), rigorously quantifying difficulty increases, and ensuring cross-model consistency. Unlike simple task expansion, BBEH substitutes original tasks with significantly more challenging variants, validated through expert annotation and automated analysis. Contribution/Results: Experiments reveal stark performance gaps: state-of-the-art general-purpose models achieve only 9.8% average accuracy, while specialized reasoning models reach 44.8%. This pronounced disparity exposes fundamental limitations in current LLMs’ high-level general reasoning—addressing a critical gap in evaluations that are both difficulty-calibrated and dimensionally comprehensive.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are increasingly deployed in everyday applications, demanding robust general reasoning capabilities and diverse reasoning skillset. However, current LLM reasoning benchmarks predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various models on BBEH and observe a (harmonic) average accuracy of 9.8% for the best general-purpose model and 44.8% for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.

Problem

• Evaluate broader reasoning proficiencies of LLMs
• Address saturation in current LLM reasoning benchmarks
• Introduce a more challenging reasoning evaluation benchmark

Innovation

• Introduces BIG-Bench Extra Hard
• Replaces tasks for increased difficulty
• Evaluates advanced reasoning capabilities