InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

2026-04-14

AI Summary

Current benchmarks for scientific reasoning are limited by publication bias, reliance on prior knowledge, label noise, and storage overhead, hindering comprehensive evaluation of large language models’ ability to reason from empirical evidence. This work proposes the first infinitely scalable, seed-driven procedural evaluation framework that generates realistic scientific directory structures and tabular data, coupled with a privileged question generator to automatically produce answerable and unanswerable questions with precise ground truth. The framework enables systematic assessment of evidence-based reasoning, refusal capability, and tool usage. Experiments reveal that current models achieve overall accuracy below 45% and consistently struggle to identify unanswerable questions; stronger models excel not by consuming more tokens, but by using tools more effectively.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 45% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.

Problem

• scientific reasoning
• benchmarking
• large language models
• empirical evaluation
• procedural generation

Innovation

• procedural generation
• scientific benchmarking
• evidence-grounded reasoning
• abstention
• tool-mediated analysis