Locally Deployable Small Language Models for Emergency Department Decision Support: A Systematic Benchmark of Fine-Tuning Strategies

2026-08-10

AI Summary

This study addresses the lack of systematic evaluation regarding patient privacy risks and fine-tuning strategies for deploying large language models in emergency departments. It presents the first comprehensive comparison of eight open-source small language models across three emergency clinical tasks—triage level prediction, specialty referral recommendation, and diagnosis prediction—using four adaptation strategies: zero-shot prompting, prefix tuning, LoRA, and full fine-tuning. Models were trained on the MIMIC-IV-ED dataset and benchmarked against commercial baselines (Claude Haiku and Sonnet 4.5). Results demonstrate that LoRA-finetuned small models outperform commercial counterparts in triage and referral tasks and successfully identify high-risk patients missed by the commercial systems, thereby validating the feasibility and clinical advantages of locally deployed models for critical decision-making.

Abstract

Deploying large language models (LLMs) for decision support in emergency departments (EDs) faces two major challenges: privacy risks of transmitting patient data to closed-source commercial LLMs and the lack of systematic evaluation of fine-tuning strategies for locally deployable open-source small language models (SLMs). We benchmarked eight open-source SLMs using zero-shot prompting, prefix tuning, Low-Rank Adaptation (LoRA), and full fine-tuning on three ED tasks: triage level prediction, specialist referral recommendation, and diagnosis prediction. Using 2,083 MIMIC-IV-ED cases and Claude Haiku 4.5 and Claude Sonnet 4.5 as baselines, we found that LoRA fine-tuned open-source SLMs outperform commercial baselines on triage level prediction and specialist referral recommendation, while diagnosis prediction remains challenging for open-source SLMs. Confusion matrix analysis further shows that fine-tuned open-source SLMs can detect highest-severity patients missed by the commercial baselines. These results demonstrate that locally deployable SLMs can achieve clinically competitive performance for ED decision support.

Problem

• emergency department
• small language models
• privacy
• fine-tuning
• decision support

Innovation

• small language models
• LoRA fine-tuning
• emergency department decision support
• local deployment
• systematic benchmarking