HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

2026-06-30

AI Summary

This work addresses the lack of systematic evaluation benchmarks for assessing end-to-end, multi-step reasoning capabilities of AI agents in realistic clinical settings. To bridge this gap, the authors introduce MedAgentBench, the first unified benchmark encompassing the full patient care continuum, featuring seven clinical workflows and 54 tasks that integrate multimodal data—including electronic health records and medical imaging—and require agents to autonomously explore the environment and execute complex decision-making with minimal instruction. The benchmark provides an interpretable overall task success rate metric, enabling end-to-end evaluation of state-of-the-art large-model agents. Experimental results reveal that even the strongest current model (Codex GPT-5.5) achieves only a 42% overall success rate, with pronounced deficiencies in medical image understanding and compositional reasoning, highlighting critical limitations of existing AI systems in complex clinical reasoning.

Abstract

As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward real-world healthcare applications. We introduce HealthAgentBench, a suite of 54 agentic healthcare tasks across 7 categories each with its unique environment. The benchmark suite spans diverse workflows throughout the patient journey and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given minimal instructions, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting. A final task success rate is reported to provide a single, interpretable metric for HealthAgentBench overall performance for each agent. Evaluating frontier agents on HealthAgentBench, we find that overall task success rate remains low, underscoring the difficulty of the suite. The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate. Beyond aggregate performance, HealthAgentBench reveals nuanced strengths and weaknesses across task categories. Frontier agents show promise in automatically developing research modeling pipelines over EHR data, but medical imaging remains especially challenging, particularly for Claude Code models, while Codex GPT-5.5 shows emerging capability. Tasks that combine large search spaces with compositional reasoning requirements remain difficult for all current agents. Together, these results suggest that HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark at https://github.com/microsoft/HealthAgentBench.

Problem

• AI agents
• healthcare benchmark
• clinical workflows
• multimodal data
• long-horizon reasoning

Innovation

• agentic healthcare benchmark
• end-to-end clinical workflow
• multi-modal medical reasoning
• long-horizon AI evaluation
• realistic healthcare environments