EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

2026-06-22

AI Summary

Existing benchmarks for evaluating medical AI agents predominantly rely on idealized data and static SQL queries, failing to capture the interactive, multi-table, and longitudinal clinical reasoning required in real-world electronic health records (EHRs). This work proposes EHR-Complex—the first large-scale interactive clinical reasoning benchmark derived from MIMIC-IV, encompassing 365K patients, 31 tables, and over 500 million records. It includes 52K tasks spanning six clinical intents, requiring agents to execute patient-level and population-level queries via SQL or Python in a sandboxed environment. EHR-Complex introduces highly complex compositional reasoning tasks (averaging 31.93 SQL components), revealing three critical failure modes in current models: flawed SQL logic, incorrect medical coding, and inadequate semantic understanding. Even the best-performing model achieves only a 62.3% exact match rate, with Pass⁴ consistency generally below 50%, underscoring significant limitations in existing approaches.

Abstract

Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.

Problem

• EHR
• clinical reasoning
• benchmark
• interactive query
• medical agents

Innovation

• EHR-Complex
• interactive clinical reasoning
• longitudinal multi-table aggregation
• compositional reasoning
• stochastic fragility