Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability

2026-07-07

AI Summary

This study explores the feasibility of using locally deployed, small open-source language models to automatically evaluate shared decision-making (SDM) in clinical settings while preserving privacy and sustainability. Grounded in the Observer OPTION12 framework, we assess multiple general-purpose and medical-domain small language models on Dutch melanoma consultation transcripts and introduce, for the first time, a Judge-LLM multi-model consensus mechanism to resolve scoring discrepancies. Experimental results indicate that general-purpose models outperform medical-specific ones, with Gemma3:12b achieving the highest correlation with human ratings (Pearson r = 0.51, Spearman ρ = 0.59). This work presents a novel, privacy-preserving, on-premises approach to automated SDM assessment under the OPTION12 framework and reveals systematic model limitations in temporal reasoning, role attribution, and evidence anchoring.

Abstract

We present LLM4SDM, the first study of open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the Observer OPTION12 framework. Unlike previous work that relies on large commercial models and the shorter OPTION5 instrument, our study focuses on privacy-preserving locally deployable models and Dutch melanoma consultation transcripts. Using expert-annotated clinical consultations, we evaluate three general-domain and two medical-domain OS-sLLMs during a development-phase pilot study. Results show that general-domain models outperform medical-domain models, which exhibit substantial hallucination and instruction-following failures. Gemma3:12b achieves the strongest agreement with human annotations (Pearson r=0.51, Spearman \r{ho}=0.59). Item-level and qualitative analyses reveal systematic challenges related to temporal discourse reasoning, conversational role attribution, and evidence grounding. We further introduce a Judge-LLM consensus framework designed to support disagreement resolution among multiple models. Our findings suggest that while current OS-sLLMs cannot replace human annotators, they offer a promising foundation for privacy-preserving human-in-the-loop SDM assessment.

Problem

• shared decision making
• OPTION12
• open-source smaller LLMs
• privacy-preserving assessment
• clinical dialogue analysis

Innovation

• open-source smaller LLMs
• shared decision making
• OPTION12
• privacy-preserving AI
• Judge-LLM consensus