ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems

2025-10-13

AI Summary

Existing academic evaluation benchmarks inadequately assess higher-order reasoning—particularly cross-disciplinary integration and comprehension of top-tier journal-level knowledge. To address this gap, we introduce AcadReason: the first benchmark explicitly designed to evaluate higher-order academic reasoning across multiple domains. It comprises 50 challenging, expert-annotated questions drawn from recent papers in leading journals across computer science, economics, law, mathematics, and philosophy. Rigorous quality control ensures annotation fidelity and task difficulty. The benchmark supports standardized evaluation under mainstream reasoning paradigms, including zero-shot and chain-of-thought prompting. We evaluate over ten state-of-the-art large language models and autonomous agent systems. Results reveal profound limitations: even the strongest model (GPT-5) scores only 16/100, while the best-performing agent achieves less than 40/100. These findings systematically expose fundamental gaps in current AI systems’ capacity for advanced, super-intelligent academic research tasks.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

In recent years, the research focus of large language models (LLMs) and agents has shifted increasingly from demonstrating novel capabilities to complex reasoning and tackling challenging tasks. However, existing evaluations focus mainly on math/code contests or general tasks, while existing multi-domain academic benchmarks lack sufficient reasoning depth, leaving the field without a rigorous benchmark for high-level reasoning. To fill this gap, we introduce the Acadreason benchmark, designed to evaluate the ability of LLMs and agents to acquire and reason over academic knowledge. It consists of 50 expert-annotated academic problems across five high-reasoning domains, including computer science, economics, law, mathematics, and philosophy. All questions are sourced from top-tier publications in recent years and undergo rigorous annotation and quality control to ensure they are both challenging and answerable. We conduct systematic evaluations of over 10 mainstream LLMs and agents. The results show that most LLMs scored below 20 points, with even the cutting-edge GPT-5 achieving only 16 points. While agents achieved higher scores, none exceeded 40 points. This demonstrates the current capability gap between LLMs and agents in super-intelligent academic research tasks and highlights the challenges of Acadreason.

Problem

• Evaluating reasoning abilities of LLMs on academic knowledge
• Addressing lack of rigorous benchmarks for high-level reasoning
• Testing models across five complex domains with expert problems

Innovation

• Introducing Acadreason benchmark for academic reasoning evaluation
• Evaluating LLMs and agents across five high-reasoning domains
• Demonstrating capability gaps in super-intelligent academic tasks