🤖 AI Summary
Existing clinical reasoning evaluation benchmarks predominantly rely on unstructured or static data, failing to capture the structured and interoperable nature of real-world electronic health records (EHRs). To address this gap, this work proposes a novel pipeline that integrates staged large language model (LLM) generation with terminology-anchored validation and repair, yielding MedCase-Structured—the first HL7 FHIR R4–compliant structured dataset for clinical reasoning assessment. Built upon the MedCaseReasoning benchmark, the pipeline successfully generates valid FHIR bundles for 82.5% of cases. Experimental results demonstrate that LLMs exhibit significantly lower diagnostic accuracy when provided with structured FHIR inputs compared to plain text, underscoring both the necessity of aligning evaluations with authentic clinical workflows and the innovative contribution of this dataset.
📝 Abstract
Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent settings remains limited. Existing benchmarks often rely on static datasets or unstructured inputs that do not reflect the structured, interoperable data formats used in clinical systems. We introduce a pipeline for generating clinically realistic HL7 FHIR R4 bundles from unstructured text, enabling controllable evaluation of clinical decision support systems. The pipeline combines staged LLM generation with terminology-grounded validation and repair to reduce hallucinated codes and enforce structural and semantic consistency. Applying this approach to MedCaseReasoning, we construct MedCase-Structured, a synthetic dataset aligned with clinician-authored diagnostic cases, achieving valid FHIR generation for 82.5% of cases. Evaluation on MedCase-Structured reveals consistently lower diagnostic accuracy for LLMs on structured FHIR inputs than with plain text, highlighting the importance of deployment-aligned benchmarking.