What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation

2026-08-24

AI Summary

为评估语言模型提出的研究想法的质量，本文引入了Lit2Test基准，通过可证伪的六领域契约来判断研究提案的质量，并对四个前沿模型进行了1200次盲评比较。

Abstract

Large language models are increasingly used to propose research ideas, yet the prevailing ways of judging such ideas supply no shared decision rule: free-form judging sways with style and position, and scoring against a later paper rewards recovery of one realized trajectory. We introduce a benchmark that carries a proposal from Literature to Test: the Lit2Test benchmark centers on a six-field contract organized around a falsifying outcome, so that every proposal precommits the observation that would prove it wrong, making its quality decidable in the first place rather than merely arguable. Built prospectively from 200 real-paper neighborhoods, Lit2Test elicits proposals from four frontier models and compares them through 1,200 pairwise comparisons judged blind in both presentation orders. The protocol audits its own reliability through diagnostic controls and bounded human calibration, with three annotators corroborating the conclusions within explicitly stated reliability bounds. Lit2Test recovers a strict ranking of the four models in all 10,000 bootstrap replicates, and the separation comes from the quality of the proposed tests and metrics rather than from surface fluency. We release the benchmark, construction pipeline, and audit artifacts for public use.

Problem

• Large language models
• Research ideas
• Judging criteria
• Falsifiable outcomes

Innovation

• Lit2Test
• falsifiable outcome
• benchmarking
• reliability audit
• research ideation