Benchmarking Motivational Interviewing Competence of Large Language Models

2026-03-04

AI Summary

This study addresses the lack of systematic evaluation of large language models’ (LLMs’) motivational interviewing (MI) capabilities in authentic clinical dialogues. For the first time, it systematically assesses the MI performance of ten open- and closed-source LLMs using the MITI 4.2 framework on both hand-crafted and real-world clinical transcripts, benchmarking them against human therapists. A double-blind experiment further evaluates psychiatrists’ ability to distinguish between human and model-generated responses. The work proposes an integrated ranking method combining MITI scores with linguistic conciseness, revealing that all models achieve competent MI proficiency, with the top-performing model significantly outperforming human experts in complex reflection ratio and evocation-to-question ratio. Psychiatrists identified machine-generated responses with only 56% accuracy, indicating highly human-like outputs and highlighting the potential of open-source models for low-resource clinical applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Motivational interviewing (MI) promotes behavioural change in substance use disorders. Its fidelity is measured using the Motivational Interviewing Treatment Integrity (MITI) framework. While large language models (LLMs) can potentially generate MI-consistent therapist responses, their competence using MITI is not well-researched, especially in real world clinical transcripts. We aim to benchmark MI competence of proprietary and open-source models compared to human therapists in real-world transcripts and assess distinguishability from human therapists. Methods: We shortlisted 3 proprietary and 7 open-source LLMs from LMArena, evaluated performance using MITI 4.2 framework on two datasets (96 handcrafted model transcripts, 34 real-world clinical transcripts). We generated parallel LLM-therapist utterances iteratively for each transcript while keeping client responses static, and ranked performance using a composite ranking system with MITI components and verbosity. We conducted a distinguishability experiment with two independent psychiatrists to identify human-vs-LLM responses. Results: All 10 tested LLMs had fair (MITI global scores >3.5) to good (MITI global scores >4) competence across MITI measures, and three best-performing models (gemma-3-27b-it, gemini-2.5-pro, grok-3) were tested on real-world transcripts. All showed good competence, with LLMs outperforming human-expert in Complex Reflection percentage (39% vs 96%) and Reflection-Question ratio (1.2 vs >2.8). In the distinguishability experiment, psychiatrists identified LLM responses with only 56% accuracy, with d-prime: 0.17 and 0.25 for gemini-2.5-pro and gemma-3-27b-it respectively. Conclusion: LLMs can achieve good MI proficiency in real-world clinical transcripts using MITI framework. These findings suggest that even open-source LLMs are viable candidates for expanding MI counselling sessions in low-resource settings.

Problem

• Motivational Interviewing
• Large Language Models
• MITI
• Clinical Transcripts
• Therapist Competence

Innovation

• Motivational Interviewing
• Large Language Models
• MITI Benchmarking
• Clinical Dialogue Simulation
• Human-AI Distinguishability