Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering

2026-05-12

AI Summary

This work addresses the challenge of integrating multi-source biomedical information—such as diseases, genes, and chemicals, with a focus on rare diseases—in multi-hop question answering. To this end, the authors construct a new dataset comprising 1,000 two-hop question-answer pairs that require models to synthesize information from two distinct Wikipedia pages. They also organize an international evaluation campaign to assess large language models’ multi-hop reasoning capabilities. The study introduces MedCPT, a novel concept-level evaluation metric, and combines retrieval-augmented generation (RAG), zero-shot reasoning, Wikipedia-based knowledge retrieval, and concept embedding matching. The best-performing system achieves 89.30% MedCPT F1 and 87.30% exact match, substantially outperforming zero-shot baselines (67.40% and 60.20%, respectively), thereby demonstrating the efficacy of the proposed retrieval strategy and advancing paradigms in both answer evaluation and system design.

Abstract

Multi-hop question answering (QA) remains a significant challenge in the biomedical domain, requiring systems to integrate information across multiple sources to answer complex questions. To address this problem, the BioCreative IX MedHopQA shared task was designed to benchmark in multi-hop reasoning for large language models (LLMs). We developed a novel dataset of 1,000 challenging QA pairs spanning diseases, genes, and chemicals, with particular emphasis on rare diseases. Each question was constructed to require two-hop reasoning through the integration of information from two distinct Wikipedia pages. The challenge attracted 48 submissions from 13 teams. Systems were evaluated using both surface string comparison and conceptual accuracy (MedCPT score). The results showed a substantial performance gap between baseline LLMs and enhanced systems. The top-ranked submission achieved an 89.30% F1 score on the MedCPT metric and an 87.30% exact match (EM) score, compared with 67.40% and 60.20%, respectively, for the zero-shot baseline. A central finding of the challenge was that retrieval-augmented generation (RAG) and related retrieval-based strategies were critical for strong performance. In addition, concept-level evaluation improved answer assessment when correct responses differed in surface form. The MedHopQA dataset is publicly available to support continued progress in this important area. Challenge materials: https://www.ncbi.nlm.nih.gov/research/bionlp/medhopqa and benchmark https://www.codabench.org/competitions/7609/

Problem

• multi-hop question answering
• biomedical domain
• information integration
• medical question answering
• complex reasoning

Innovation

• multi-hop reasoning
• retrieval-augmented generation
• biomedical question answering
• concept-level evaluation
• rare diseases