RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation

2026-05-13

AI Summary

Existing ICU evaluation benchmarks treat physicians’ historical actions as ground-truth labels, yet these actions are based on incomplete information and thus inadequately assess AI systems’ clinical reasoning capabilities. This work proposes RealICU—the first ICU benchmark grounded in retrospective annotations by senior clinicians over complete patient trajectories—featuring four clinical tasks evaluated within a 30-minute window to probe large language models’ (LLMs’) long-context comprehension and decision-making. The study reveals a trade-off between recall and safety in current LLMs, including memory-augmented variants, alongside evidence of anchoring bias. To address these limitations, the authors introduce ICU-Evo, a structured memory agent designed to explore improved reasoning pathways. Concurrently, they release RealICU-Gold (930 cases) and RealICU-Scale (11,862 cases), establishing a new testbed for high-stakes clinical decision-making.

Abstract

Intensive care units (ICU) generate long, dense and evolving streams of clinical information, where physicians must repeatedly reassess patient states under time pressure, underscoring a clear need for reliable AI decision support. Existing ICU benchmarks typically treat historical clinician actions as ground truth. However, these actions are made under incomplete information and limited temporal context of the underlying patient state, and may therefore be suboptimal, making it difficult to assess the true reasoning capabilities of AI systems. We introduce RealICU, a hindsight-annotated benchmark for evaluating large language models (LLMs) under realistic ICU conditions, where labels are created after senior physicians review the full patient trajectory. We formulate four physician-motivated tasks: assess Patient Status, Acute Problems, Recommended Actions, and Red Flag actions that risk unsafe outcomes. We partition each trajectory with 30-min windows and release two datasets: RealICU-Gold with 930-window annotations from 94 MIMIC-IV patients, and RealICU-Scale with 11,862 windows extended by Oracle, a physician-validated LLM hindsight labeler. Existing LLMs including memory-augmented ones performed poorly on RealICU, exposing two failure modes: a recall-safety tradeoff for clinical recommendations, and an anchoring bias to early interpretations of the patient. We further introduce ICU-Evo to study structured-memory agents that improves long-horizon reasoning but does not fully eliminate safety failures. Together, RealICU provides a clinically grounded testbed for measuring and improving AI sequential decision-support in high-stakes care. Project page: https://chengzhi-leo.github.io/RealICU-Bench/

Problem

• ICU
• long-context reasoning
• clinical decision support
• benchmark
• LLM evaluation

Innovation

• RealICU
• hindsight annotation
• long-context reasoning
• clinical decision support
• structured-memory agents