IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs

2026-07-01

AI Summary

This study addresses the challenge of disentangling the contributions of reasoning capability and domain knowledge retrieval in large language models (LLMs) for scientific reasoning. To this end, the authors construct a benchmark of isomorphic cross-disciplinary scientific problem pairs that preserve identical logical structures while varying the required domain knowledge, enabling the first controlled evaluation that decouples reasoning ability from knowledge dependence. Through multi-model comparisons, confidence interval analysis, and validation against established benchmarks such as GPQA Diamond, the findings reveal that 91.3% of observed reasoning gains stem from domain knowledge rather than generalizable reasoning mechanisms. Moreover, activating specialized reasoning modules in high-capability models yields less than a 5-percentage-point improvement, and current dedicated reasoning models significantly underperform standard LLMs on this benchmark.

Abstract

We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in LLM evaluation. Each pair shares identical logical structure but requires different domain-specific knowledge, enabling controlled attribution of reasoning-mode gains. Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving. Reasoning toggles on highly capable models provide less than 5 percentage points accuracy gain across all domains, and a reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA Diamond (+19.2 percentage points) underperforms on ISOSCI (-24.7 percentage points), showing that benchmark choice determines conclusions about reasoning utility. We release ISOSCI at https://huggingface.co/datasets/isosci/isosci

Problem

• reasoning
• knowledge retrieval
• large language models
• scientific problem-solving
• evaluation benchmark

Innovation

• isomorphic benchmark
• reasoning evaluation
• knowledge retrieval
• cross-domain science problems
• LLM reasoning