mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health

2026-06-28

AI Summary

This work addresses the lack of coverage in existing medical question-answering benchmarks for midwife-relevant topics in maternal, neonatal, and reproductive health, as well as the absence of publicly available fine-grained retrieval relevance evaluation datasets. To bridge this gap, the authors introduce two novel benchmarks: mamabench, comprising 25,949 diverse questions derived from expert sources, and mamaretrieval, featuring 3,185 clinical queries annotated with fine-grained (0–6) relevance scores against 63,650 guideline text chunks. The study innovatively leverages original expert-formulated questions and continuous relevance scoring to distinguish between responses that directly answer a query versus those merely topically related. Rigorous quality assessments—including scope consistency, state-of-the-art discriminability, and pool completeness—are systematically conducted. These benchmarks constitute the first high-quality, open-source evaluation resources for retrieval-augmented generation systems in maternal health and have already been deployed to evaluate real-world edge-side clinical assistants.

Abstract

Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline retrieval. We release two benchmarks that fill these gaps. mamabench is a scope-filtered QA set of 25,949 items assembled from seven existing expert-authored sources across multiple-choice, short-answer, and rubric-graded tracks; to help users calibrate the LLM judge that scores the rubric track, we re-scope HealthBench's physician-labelled meta-evaluation to the domain. mamaretrieval pairs 3,185 clinical queries with graded (0-6) relevance labels over a 63,650-chunk maternal-health guideline corpus, using a decomposed rubric that distinguishes a chunk that answers a query from one merely on its topic. Three decisions shape both: assemble and filter expert sources rather than author questions, grade relevance rather than binarise it, and measure and disclose the limits of the labels -- scope-classifier agreement, a frontier-judge check, and a pooling-completeness audit -- rather than treat them as an oracle. A companion paper uses the benchmarks to evaluate a deployed on-device assistant; both are released openly for research.

Problem

• medical question-answering
• maternal health
• retrieval-augmented generation
• benchmark
• relevance evaluation

Innovation

• retrieval-augmented generation
• maternal health benchmark
• graded relevance labeling
• expert-sourced QA
• transparent evaluation