MeNTi: Bridging Medical Calculator and LLM Agent with Nested Tool Calling

2024-10-17

arXiv.org

AI Summary

This work addresses the limited tool-calling capability of large language models (LLMs) in quantitative clinical assessment. We propose a medical-calculator-oriented agent architecture. Methodologically, we introduce a novel meta-tool abstraction and nested function invocation mechanism to enable dynamic composition of medical toolchains; further, we integrate medical knowledge-constrained decoding with multi-layer semantic parsing to jointly resolve calculator selection, parameter slot filling, and unit conversion. We construct CalcQA—a domain-specific benchmark comprising 100 real-world clinical cases and 281 medical calculators. Experiments demonstrate that our architecture significantly outperforms existing baselines in accuracy, robustness, and clinical interpretability. It establishes a new paradigm for deploying LLMs in high-precision, verifiable medical computation scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Integrating tools into Large Language Models (LLMs) has facilitated the widespread application. Despite this, in specialized downstream task contexts, reliance solely on tools is insufficient to fully address the complexities of the real world. This particularly restricts the effective deployment of LLMs in fields such as medicine. In this paper, we focus on the downstream tasks of medical calculators, which use standardized tests to assess an individual's health status. We introduce MeNTi, a universal agent architecture for LLMs. MeNTi integrates a specialized medical toolkit and employs meta-tool and nested calling mechanisms to enhance LLM tool utilization. Specifically, it achieves flexible tool selection and nested tool calling to address practical issues faced in intricate medical scenarios, including calculator selection, slot filling, and unit conversion. To assess the capabilities of LLMs for quantitative assessment throughout the clinical process of calculator scenarios, we introduce CalcQA. This benchmark requires LLMs to use medical calculators to perform calculations and assess patient health status. CalcQA is constructed by professional physicians and includes 100 case-calculator pairs, complemented by a toolkit of 281 medical tools. The experimental results demonstrate significant performance improvements with our framework. This research paves new directions for applying LLMs in demanding scenarios of medicine.

Problem

• Enhance LLM tool utilization in medicine
• Address medical calculator selection issues
• Develop benchmark for LLM clinical assessments

Innovation

• Nested tool calling mechanism
• Specialized medical toolkit integration
• Meta-tool for flexible selection