MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

2026-07-02

AI Summary

Existing medical computation benchmarks struggle to handle complex real-world clinical scenarios involving multi-calculator coordination, nested invocations, and ambiguous queries. To address this gap, this work introduces MedCalc-Pro, a benchmark comprising 2,268 real clinical cases spanning 77 medical calculators across 14 specialties, and proposes the first three-tiered progressive evaluation framework tailored for complex medical computations. Furthermore, the authors design a general-purpose LLM agent architecture capable of multi-tool selection and nested calculator calls, enhanced with structured parameter validation and evidence verification mechanisms to effectively mitigate error propagation. Experimental results demonstrate that the proposed approach significantly outperforms existing open-source, closed-source, and medical-domain-specific large language models across single-calculator, multi-calculator, and nested-task settings.

Abstract

Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case corresponds to a single calculator and the required tool is explicitly specified in the query. However, real clinical scenarios often require multiple calculators for joint evaluation, nested-scale calculation, and fuzzy queries that do not directly specify the target calculator. To this end, we propose a new medical calculation benchmark, MedCalc-Pro, which covers three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator calculation settings. MedCalc-Pro contains 2,268 real-world clinical cases, covering 77 medical calculators across 14 clinical departments. Meanwhile, to address the limited performance of existing frameworks and methods in complex clinical scenarios, we further propose a more generalizable agent framework that supports multi-tool selection and nested-tool calling, while suppressing parameter error propagation through structured validation and evidence review. We conduct systematic comparisons across open-source, closed-source, and medical-specialized LLMs, and the results show that our framework achieves the best performance across all three task settings. This work provides a new benchmark and method for evaluating and applying LLMs in challenging medical calculation scenarios.

Problem

• medical calculation
• LLM benchmark
• multi-calculator
• nested calculation
• fuzzy query

Innovation

• medical calculation benchmark
• LLM agent framework
• multi-tool selection
• nested-tool calling
• error propagation suppression