Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot

2026-08-17

AI Summary

This study addresses the subjectivity and lack of objective standards in evaluating scientific question generation by proposing a historical backtesting protocol based on temporally isolated corpora. Through a model-agnostic, reproducible framework with prospectively frozen test sets, validated via astronomical case studies, we demonstrate that evidence-structure-prioritized generation significantly outperforms pure LLM prompting. Furthermore, low human-AI agreement reveals the unreliability of high inter-model consensus. By releasing a reproducible astronomy dataset and prospective test instances for 2026–2030, this work establishes an objective, contamination-free paradigm for automated evaluation, effectively advancing standardization in the field.

Abstract

Systems that generate scientific research questions are evaluated today by expert scores, LLM-as-judge ratings, or curated case studies -- all subjective, none falsifiable. We formalize historical backtesting as an alternative: a system generates questions from a corpus frozen at a historical cutoff, the questions are frozen before any access to later literature, and a temporally isolated future corpus then determines whether each question was subsequently answered, partially addressed, independently posed, or ignored, and whether its underlying premise was supported or refuted. The protocol is model-agnostic: any system that emits frozen questions can be scored. We release reproducible astronomy instances with temporally isolated corpora, frozen questions, auditable labels, four reference baselines, and a submission interface. Two findings result. First, evidence-structure-first generation outperforms LLM-only prompting: across a generator decomposition crossed with a four-cutoff stress test (2010-2024, 798 judged questions) whose last window postdates model training, LLM-only generation shows memorized relevance without specific foresight, while a generator using no model weights at all finds questions whose premises the future refutes in every era. Second, a seven-rater agreement study (two blinded human annotators, five judge models, 90 items) indicts the outcome taxonomy rather than the judge: two careful humans agree at kappa = 0.17, every judge model agrees with the professional annotator as well or better (0.17-0.26), and frontier models agree with one another at 0.60 -- certifying an LLM judge by model-model agreement would have overstated its reliability threefold. A prospective instance -- 200 questions frozen 2026-08-17, scored 2027-2030 -- is released so the central claims become contamination-free tests that time itself will grade.

Problem

• Scientific Question Discovery
• Evaluation Protocol
• Historical Backtesting
• Falsifiability
• Subjectivity

Innovation

• Historical Backtesting
• Scientific Question Discovery
• Evidence-structure-first Generation
• Temporal Isolation
• LLM-as-judge Reliability