Where do LLMs Fall Short in CBT-Guided Affective Reasoning?

2026-07-02

AI Summary

Although large language models possess knowledge of cognitive behavioral therapy (CBT), they tend to default to empathetic and validating responses in dialogue, struggling to flexibly deploy CBT strategies. To address this, this work proposes a knowledge-guided framework that parses user utterances into Beck’s cognitive conceptualization structure, integrates the SNOMED CT clinical ontology with natural language inference, and dynamically selects intervention strategies—such as Socratic questioning or alternative perspective-taking—via a Multi-Chain-of-Thought (MCoT) mechanism. The study introduces Protocol Leverage Force (F), a novel behavior-level metric that quantifies the actual behavioral shift induced by CBT prompting. Experimental results demonstrate that MCoT significantly outperforms single-chain prompting; however, the observed behavioral shift remains modest at only 1.2–1.3%, revealing a critical gap between possessing CBT knowledge and effectively applying it.

Abstract

Cognitive Behavioral Therapy (CBT) provides a structured framework for understanding a user's mental state by examining the interaction between cognitive and behavioral factors. However, out-of-the-box LLMs respond fluently and empathetically, yet collapse into validation & reflection, regardless of what the user actually needs. They know theoretical CBT (scoring up to 96% accuracy on licensing exam questions) but fail to apply it effectively. We explore this gap with a knowledge-guided framework that treats CBT dialogue as controlled affective reasoning: user narratives are decomposed into Beck's Cognitive Conceptualization structure, grounded in clinical SNOMED CT concepts validated via Natural Language Inference, and a Multiple Chain-of-Thought (MCoT) strategy selection between Validation & Reflection, Socratic Questioning, or Alternative Perspectives. To measure whether such guidance actually changes behavior, we introduce the Protocol Leverage Force (F), a behavior-level metric that captures how far an intervention shifts a model away from its default response. Across three open-weight LLMs and 14 RealCBT-derived case studies, evaluated with human experts, valence-arousal trajectories, and linguistic entrainment, F shows that simply introducing protocol definitions via single chain-of-thought prompting fails to change LLM behavior, while MCoT on these definitions guides strategy selection better. Still, the effect stays within 1% (approx. 1.2-1.3%), and all models remain biased toward Validation & Reflection. These results show CBT knowledge alone does not ensure effective application, giving the affective-computing community instrumentation to measure where LLMs fall short.

Problem

• LLMs
• Cognitive Behavioral Therapy
• affective reasoning
• protocol adherence
• behavioral bias

Innovation

• Multiple Chain-of-Thought
• Protocol Leverage Force
• Affective Reasoning
• SNOMED CT
• Cognitive Conceptualization