CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR

2026-08-07

AI Summary

This work addresses the gap between strong performance of large language models on medical benchmarks and their limited capacity for defensible reasoning in real-world clinical settings using heterogeneous, longitudinal electronic health records (EHRs). To bridge this gap, the authors introduce CliniCARE-Bench—the first benchmark designed for clinical auditing—comprising 25 physician-validated scenarios and 750 real-world cases derived from MIMIC-IV. Agents must integrate structured and unstructured EHR data within a constrained tool environment to issue one of four policy-compliant rulings while fully tracing their investigative process. The framework uniquely unifies longitudinal EHR interrogation, evidence provenance, policy adherence, procedural compliance, and a calibrated abstention mechanism that distinguishes “missing data” from “medical ambiguity.” Reference rulings are established via multi-model arbitration and clinical committee calibration. Evaluation across 16 systems shows four-class accuracy of 65.3%–76.1%, yet defect-free accuracy—excluding shortcut-based responses—drops markedly by 4.8–14.8 percentage points, revealing substantial overestimation by conventional metrics.

Abstract

Large language models perform strongly on medical knowledge benchmarks, but reliable clinical deployment requires agents to conduct defensible investigations over heterogeneous, longitudinal records: determining what evidence is needed, retrieving and reconciling structured and free-text data, grounding conclusions in verifiable evidence, and deferring cases that cannot be resolved reliably. We introduce CliniCARE-Bench (Clinical Calibrated Audit of Medical Reasoning in EHR), a benchmark for retrospective clinical audit: 25 clinician-validated scenarios instantiated as 750 patient-specific cases over real-patient-derived MIMIC-IV data. Systems investigate each case through a governed, logged tool environment for record retrieval, computation, and policy access, and return one of four verdicts---Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous---the last two separating missing evidence from residual medical ambiguity. Beyond verdict accuracy, we score patient-evidence and policy grounding, process adherence, calibrated abstention, reliability, and efficiency against case-level reference verdicts produced by independent multi-model adjudication and calibrated against Clinical Board review. Every retrieval, computation, and report is replayable, so the investigation trace is inspectable and scorable. To our knowledge, CliniCARE-Bench is the first deployment-oriented clinical-agent benchmark to jointly evaluate real longitudinal EHR investigation, claim-level evidence grounding, governing-policy use, process adherence, and calibrated abstention within a common patient-level adjudication framework. Across 16 agentic systems, four-way accuracy spans 65.3-76.1%, but raw accuracy overstates investigation quality. Defect-free accuracy, which credits a verdict only when correct and free of prohibited shortcuts, is 4.8-14.8 points lower and reorders the leaderboard.

Problem

• clinical reasoning
• electronic health records
• medical audit
• evidence grounding
• calibrated abstention

Innovation

• CliniCARE-Bench
• clinical reasoning
• electronic health records (EHR)
• calibrated abstention
• evidence grounding