SFBench: The SciFy Scientific Feasibility Benchmark

2026-06-28

AI Summary

This study addresses the lack of high-quality benchmarks for evaluating models’ ability to assess the feasibility of scientific claims. To this end, we introduce a novel benchmark dataset comprising 197 original materials science claims, each annotated by domain experts with a five-point feasibility rating and an open-ended natural language explanation. This benchmark uniquely combines non-literature-derived claims, expert annotation, and structured scoring—a design that substantially mitigates training data contamination risks while increasing task complexity. Using this dataset, we conduct baseline evaluations with GPT-family models, revealing significant limitations in current large language models’ capacity for complex scientific reasoning. Our work establishes a robust foundation for future research in scientifically grounded model evaluation.

Abstract

We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in materials science, each annotated with a ground-truth feasibility score on a five-point scale along with an explanation of that assessment. The collection differs from previous collections in several important ways: 1) it defines a complex task that requires reasoning over claims of varying scientific feasibility; 2) its claims are not extracted from existing scientific publications but are created de novo, greatly reducing the chances that LLMs have trained on them; 3) claims and ground truth are established by subject matter experts, not by artificial intelligence; and 4) unlike many benchmarks that ask about question/answer pairs, provide multiple choice answers, or ask questions requiring short, fixed answers, SFBench explanations are completely open-ended. We describe the benchmark design, data creation process, and evaluation metrics, and we report baseline results using recent GPT models.

Problem

• scientific feasibility
• benchmark dataset
• materials science
• claim evaluation
• LLM evaluation

Innovation

• scientific feasibility
• benchmark dataset
• de novo claim generation
• expert annotation
• open-ended explanation