MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering

2026-05-12

AI Summary

Existing biomedical question-answering benchmarks struggle to effectively evaluate the multi-hop reasoning capabilities of large language models, often suffering from answer leakage, performance saturation, and data contamination. To address these limitations, this work introduces a disease-centric multi-hop QA benchmark that requires models to synthesize information from two Wikipedia articles to generate open-ended answers. The authors propose a reusable dataset construction framework featuring a novel structured validation pipeline that combines expert annotation with LLM-as-a-judge evaluation. Semantic enrichment and concept-level assessment are enabled through integration of the MONDO, NCBI Gene, and NCBI Taxonomy ontologies, while a large-scale question bank with hidden answers enhances robustness against data contamination. The released benchmark comprises 1,000 high-quality questions (embedded within a publicly available set of 10,000) and supports the BioCreative IX shared task, significantly improving discriminative power for assessing genuine model reasoning abilities.

Abstract

Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish reasoning from pattern matching and remain discriminative as model capabilities improve. Existing biomedical question answering (QA) benchmarks are limited in this respect. Multiple-choice formats can allow models to succeed through answer elimination rather than inference, while widely circulated exam-style datasets are increasingly vulnerable to performance saturation and training data contamination. Multi-hop reasoning, defined as the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnostic support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks. MedHopQA is a disease-centered multi-hop reasoning benchmark consisting of 1,000 expert-curated question-answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in an open-ended free-text format. Gold annotations are augmented with ontology-grounded synonym sets from MONDO, NCBI Gene, and NCBI Taxonomy to support both lexical and concept-level evaluation. MedHopQA was constructed through a structured process combining human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1,000 scored questions are embedded within a publicly downloadable set of 10,000 questions, with answers withheld, on a CodaBench leaderboard. MedHopQA provides both a benchmark and a reusable framework for constructing future biomedical QA datasets that prioritize compositional reasoning, saturation resistance, and contamination resistance as core design constraints.

Problem

• biomedical question answering
• multi-hop reasoning
• large language models
• benchmark evaluation
• reasoning vs pattern matching

Innovation

• multi-hop reasoning
• biomedical question answering
• ontology-grounded evaluation
• contamination-resistant benchmark
• LLM-as-a-judge