OpenBioRQ: Unsolved Biomedical Research Questions for Agents

2026-06-20

AI Summary

This work addresses the critical issue that existing biomedical agents often generate responses inconsistent with cited literature, a flaw overlooked by conventional evaluations reliant on fixed-answer benchmarks. To tackle this, the authors introduce the first retrieval-based agent evaluation benchmark for open-ended biomedical questions, comprising 12,553 unresolved queries spanning 12 domains. Agents must perform multi-turn tool-augmented reasoning and autonomously decide when to abstain from answering, enabling assessment of both factual faithfulness and refusal capability. Key innovations include an open-question paradigm without predefined answers, validation of question openness via subsequent real-world literature, objective difficulty calibration based on reference model failure rates, and a frozen checklist that significantly improves annotation consistency (Spearman correlation rising from 0.35 to 0.82). Experiments reveal that even state-of-the-art agents achieve only 29%–60% success on the hardest subset and exhibit pronounced tool-use degradation under high difficulty.

Abstract

A working citation looks like proof -- but the fact that a link resolves does not mean the cited paper supports the claim. I find that current agentic models rarely fabricate citations (over $99\%$ resolve), yet roughly $15.9\%$ link to the wrong paper. Existing benchmarks miss this failure mode: when a question has a fixed answer key, a model can reproduce the expected source from that key rather than independently verifying that the source supports the claim. I introduce \textbf{\openbiorq{}}, a retrieval-grounded agentic benchmark of $12{,}553$ unsolved biomedical research questions across $12$ domains that treats open questions as a faithfulness-and-abstention probe. To my knowledge, this is the first biomedical benchmark to combine an agentic setting -- where the model must issue multiple tool calls -- with unsolved questions that have no answer key. Openness is verified against real follow-up evidence rather than a model's parametric knowledge. Difficulty is empirical: I anchor it on questions that three open-weight reference models fail to answer, rather than on subjective hardness labels. On this hardest subset, held-out models from the same lineage as the difficulty anchors solve only ~17%, while three independent frontier agents (Gemini-3-Pro, Opus-4.7, GPT-5.5) span a wide 29-60% range. The benchmark is thus hard, non-saturating (the best agent still leaves ~33-40\% unsolved), and discriminating across capability tiers. Beyond difficulty, I observe agentic collapse on the hardest questions, where agents stop using their tools. For the most collapse-prone model, blocking tool access entirely barely changes its score -- so tools stop paying off exactly where they are needed most. A frozen per-question checklist raises inter-judge agreement from Spearman 0.35 to 0.82.

Problem

• biomedical research questions
• citation faithfulness
• agentic benchmarks
• open-ended questions
• retrieval grounding

Innovation

• agentic benchmark
• unsolved biomedical questions
• faithfulness evaluation
• tool-use collapse
• retrieval-grounded reasoning