EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries

2026-06-14

AI Summary

Existing clinical question-answering benchmarks struggle to evaluate models’ ability to perform multi-turn, evidence-based reasoning over longitudinal electronic health records using multiple discharge summaries. To address this gap, this work introduces the first multi-turn, evidence-driven clinical QA benchmark specifically designed for longitudinal discharge summaries. The benchmark leverages structured parsing, expert-designed question templates, large language model generation, and rigorous manual validation by medical professionals to construct a high-quality dataset spanning eight clinical topics. It comprises 967 patients and 16,072 expert-verified question-answer pairs, with each answer explicitly grounded in textual evidence. The dataset uniquely pairs content questions with corresponding evidence localization tasks. Evaluation of 22 state-of-the-art large language models reveals significant deficiencies in evidence grounding and multi-turn consistency.

Abstract

Discharge summaries are crucial clinical documents containing the context of a patient's overall hospital stay, and are routinely reviewed by medical experts for patient readmission, ongoing care, and diagnostic decision-making. When reviewing them, medical experts often must iteratively synthesize information across multiple summaries while verifying the evidence supporting each answer. Although large language models (LLMs) are increasingly explored for clinical question answering, existing benchmarks do not sufficiently reflect this setting: they often evaluate exam-style medical knowledge or focus on single-turn question answering with limited evidence-grounding evaluation. We introduce EHRNote-ChatQA, the first benchmark for evidence-grounded multi-turn clinical question answering over patients' multiple discharge summaries. Built from de-identified MIMIC-IV discharge summaries, EHRNote-ChatQA contains 967 patient-level multi-turn samples spanning one to five notes and 16,072 medical-expert-verified QA pairs (8,036 content questions, each paired with an evidence-grounding question) across eight clinical categories. The benchmark is constructed through an expert-informed pipeline combining discharge-summary structuring schema, expert-curated multi-turn QA templates, and LLM-based generation, followed by review and revision of every single QA sample by 11 medical experts. Benchmarking 22 open- and closed-source LLMs reveals several challenges, including that LLMs struggle more with evidence grounding than content answering, multi-turn errors compound across turns, and single-turn clinical QA performance does not reliably transfer to this setting. These findings establish EHRNote-ChatQA as a rigorous and practical benchmark for evaluating clinical QA systems. The dataset will be made publicly available through PhysioNet credentialed access.

Problem

• clinical question answering
• evidence grounding
• multi-turn dialogue
• discharge summaries
• longitudinal EHR

Innovation

• evidence-grounded QA
• multi-turn clinical dialogue
• longitudinal discharge summaries
• expert-validated benchmark
• clinical LLM evaluation