Clinical Term Extraction using Open-Source Small Language Models

2026-06-19

AI Summary

This study addresses the challenge of extracting structured information on amyotrophic lateral sclerosis (ALS) from unstructured clinical notes, which are otherwise difficult to analyze directly. It presents the first systematic evaluation of 26 open-source small language models (SLMs) for zero-shot extraction of 17 ALS-related clinical concepts from discharge summaries, using few-shot prompting combined with structured JSON templates. The authors propose a hybrid extraction pipeline that integrates rule-based and model-based approaches and evaluate performance through multi-label metrics and manual validation. Results indicate that regular expression rules achieve higher micro-F1 scores and lower Hamming loss overall, while among SLMs, Qwen3-4B-Instruct-2507 performs best overall, and Hammer2.1-7b excels specifically in detecting ALSFRS-R subscale items.

Abstract

Clinical information for amyotrophic lateral sclerosis (ALS) care documented in unstructured clinical notes limits downstream analysis without extraction into structured formats. Open-source small language models with few-shot prompting for detecting the presence of ALS-relevant clinical terms in patient documentation were evaluated without task-specific training data. The detection task targeted 17 categories spanning functional scores, respiratory measures, medications, and related clinical and non-clinical attributes. Clinical note content was normalized from JSON-encoded discharge summaries and processed with a prompt template having structured JSON outputs. We compared 26 open-source models using aggregate, label-level, and manual-validation multilabel classification metrics. Manual validation showed that a regex rule baseline had higher overall micro-F1 and lower Hamming loss than any single SLM or TF-IDF baseline, while Qwen3-4B-Instruct-2507 was the highest-performing SLM by micro-F1. Model rankings varied by metric and label category, with the TF-IDF baseline showing high recall but low precision, some SLMs showing higher precision but lower recall, and Hammer2.1-7b showing strong performance for ALSFRS-R subscore detection. These findings support targeted hybrid extraction workflows rather than replacement of existing rule-based methods.

Problem

• Clinical Term Extraction
• Amyotrophic Lateral Sclerosis
• Unstructured Clinical Notes
• Structured Data Conversion
• Natural Language Processing

Innovation

• small language models
• few-shot prompting
• clinical term extraction
• structured output prompting
• hybrid NLP workflow