Comparisons between a Large Language Model-based Real-Time Compound Diagnostic Medical AI Interface and Physicians for Common Internal Medicine Cases using Simulated Patients

2025-05-27

AI Summary

This study addresses the low diagnostic accuracy, prolonged duration, and high cost of initial diagnoses in primary care. We propose a real-time composite diagnostic AI interface powered by large language models (LLMs). Methodologically, we introduce a novel real-time interactive architecture that integrates multi-stage clinical reasoning with dynamic information assimilation, grounded in USMLE Step 2 CS–style case modeling and simulated patient assessment. Experimental results demonstrate that the AI achieves 80% top-1 diagnostic accuracy—surpassing physicians’ 50–70%—and 100% top-2 accuracy—exceeding physicians’ 70–90%. Diagnostic time is reduced by 44.6%, and per-case cost drops by 98.1%; patient satisfaction matches that of physicians. To our knowledge, this is the first LLM-based system achieving clinical-grade real-time composite diagnosis, delivering concurrent advances in diagnostic accuracy, operational efficiency, and economic viability—establishing a deployable technical paradigm for AI-augmented primary-care triage.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Objective To develop an LLM based realtime compound diagnostic medical AI interface and performed a clinical trial comparing this interface and physicians for common internal medicine cases based on the United States Medical License Exam (USMLE) Step 2 Clinical Skill (CS) style exams. Methods A nonrandomized clinical trial was conducted on August 20, 2024. We recruited one general physician, two internal medicine residents (2nd and 3rd year), and five simulated patients. The clinical vignettes were adapted from the USMLE Step 2 CS style exams. We developed 10 representative internal medicine cases based on actual patients and included information available on initial diagnostic evaluation. Primary outcome was the accuracy of the first differential diagnosis. Repeatability was evaluated based on the proportion of agreement. Results The accuracy of the physicians' first differential diagnosis ranged from 50% to 70%, whereas the realtime compound diagnostic medical AI interface achieved an accuracy of 80%. The proportion of agreement for the first differential diagnosis was 0.7. The accuracy of the first and second differential diagnoses ranged from 70% to 90% for physicians, whereas the AI interface achieved an accuracy rate of 100%. The average time for the AI interface (557 sec) was 44.6% shorter than that of the physicians (1006 sec). The AI interface ($0.08) also reduced costs by 98.1% compared to the physicians' average ($4.2). Patient satisfaction scores ranged from 4.2 to 4.3 for care by physicians and were 3.9 for the AI interface Conclusion An LLM based realtime compound diagnostic medical AI interface demonstrated diagnostic accuracy and patient satisfaction comparable to those of a physician, while requiring less time and lower costs. These findings suggest that AI interfaces may have the potential to assist primary care consultations for common internal medicine cases.

Problem

• Comparing AI and physicians in diagnosing common internal medicine cases
• Evaluating accuracy and speed of AI versus human doctors
• Assessing cost-effectiveness of AI in primary care diagnostics

Innovation

• LLM-based real-time diagnostic AI interface
• Clinical trial comparing AI and physicians
• AI reduces diagnosis time and cost