Toward Trustworthy Large Language Model Agents in Healthcare

2026-07-06

AI Summary

This study addresses the inefficiencies of manual coordination, system fragmentation, and high administrative costs in medical appointment scheduling by proposing CareConnect—a fully automated system leveraging large language model function calling, retrieval-augmented generation (RAG), and domain-specific tool orchestration. The work introduces a novel safety architecture that integrates deterministic short-circuit mechanisms with scope constraints to ensure operational reliability and regulatory compliance, explicitly prohibiting any form of diagnostic or therapeutic advice. Evaluated across 680 test scenarios, CareConnect achieves a 91.8% task completion rate and a 96.0% safety compliance rate, with a per-appointment cost of merely $0.0324—significantly outperforming conventional human-driven scheduling approaches.

Abstract

Healthcare appointment scheduling remains a persistent operational bottleneck, driven by manual coordination, fragmented legacy systems, and high administrative overhead. These inefficiencies constrain provider availability and degrade patient access to care. This paper presents CareConnect, a safety-first conversational agent for healthcare logistics automation that leverages large language model (LLM) function calling, retrieval-augmented generation (RAG), and layered deterministic safety guardrails. The system orchestrates eight domain-specific tools to support appointment booking, modification, cancellation, and facility information retrieval, while enforcing strict scope constraints that prohibit medical advice or diagnosis. Safety-critical situations are handled through deterministic short-circuit mechanisms for emergency detection and medical intent refusal. We evaluate CareConnect on a comprehensive benchmark of 680 task-oriented scenarios spanning end-to-end workflows, multi-turn interactions, and edge cases. Experimental results demonstrate a 91.8% task completion rate with a median per-request latency of 2.2 seconds, 96.0% safety compliance on the dedicated safety-critical evaluation subset, and an average operational cost of $0.0324 per appointment, yielding a significant cost reduction compared to manual human scheduling. These findings show that carefully scoped and rigorously safeguarded LLM-based agents can reliably automate complex healthcare operational workflows while maintaining safety guarantees and achieving substantial cost efficiency. The source code and system implementation are publicly available at https://github.com/Hadi-Hsn/CareConnect.

Problem

• healthcare appointment scheduling
• operational bottleneck
• administrative overhead
• patient access
• legacy systems

Innovation

• large language model agents
• retrieval-augmented generation
• deterministic safety guardrails
• healthcare automation
• function calling