LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making

2026-07-10

AI Summary

Current large language models in medical evaluation are largely confined to short-context question answering and tool invocation, struggling to address the longitudinal decision-making demands inherent in real-world clinical practice spanning multiple patient encounters. This work proposes LongMedBench—the first benchmark specifically designed for long-horizon clinical decision-making—constructed from MIMIC-IV to form temporally structured event streams and long-context memory datasets that enable agent-based longitudinal interaction across multi-session scenarios. The benchmark encompasses three task types: factual question answering, temporal reasoning, and long-horizon decision-making, accompanied by a reproducible data construction pipeline. Experiments reveal that while existing models effectively leverage explicit timestamps, they underperform on implicit temporal reasoning; although retrieval-augmented generation (RAG) and memory mechanisms improve information retrieval, long-horizon decisions remain constrained by overreliance on immediate context.

Abstract

In this work, we introduce LongMedBench, a real-world EHR-based benchmark for long-horizon clinical decision-making. Prior evaluations of LLM-based medical agents have largely emphasized short-context knowledge QA and tool use. However, real-world medical care is inherently longitudinal, and clinicians must aggregate evidence across repeated visits, tests, and evolving treatments. Therefore, long-horizon interaction is essential for realistic assessment. LongMedBench is constructed via a reproducible pipeline that integrates MIMIC-IV admission records and clinical notes into time-series event streams and long-context memory datasets, enabling long-horizon, multi-session interactions between agents and a clinical environment. It comprises 335 patients, with 19.72 inpatient visits per patient on average and 44.91 medical events per visit. Guided by the long-horizon decision process, we propose an evaluation taxonomy with three suites: fact-based QA, temporal reasoning, and long-horizon decision-making. This taxonomy measures how agents understand and leverage historical patient information over extended horizons. Our experiments show that while recent LLMs can make good use of explicit timestamps, they have challenges in implicit time inference; The RAG and agent memory system can improve the performance of information retrieval tasks, but the performance of decision-making tasks is highly dependent on the model's immediate context.

Problem

• long-horizon clinical decision-making
• medical agents
• EHR-based benchmark
• temporal reasoning
• longitudinal care

Innovation

• Long-horizon clinical decision-making
• Medical agent benchmarking
• Electronic Health Records (EHR)
• Temporal reasoning
• Long-context memory