Can You Share Your Story? Modeling Clients' Metacognition and Openness for LLM Therapist Evaluation

2025-07-25

AI Summary

Existing LLM-based therapist evaluation methods rely on client simulators that explicitly disclose internal states, failing to assess deep understanding of unexpressed beliefs and thoughts. To address this, we propose MindVoyager—the first controllable client simulation framework supporting dynamic mental modeling and metacognitive simulation. It integrates psychological theory, dialogue strategy control, and LLM-driven multi-turn interaction to progressively reveal and adaptively evolve clients’ implicit cognitive-affective states. Crucially, it introduces novel constructs: client openness and metacognitive modeling—enabling fine-grained, behaviorally grounded evaluation metrics. This facilitates the first dynamic, ecologically valid assessment of LLM therapists’ capacity for probing latent mental states. Experiments demonstrate that MindVoyager significantly discriminates model performance on implicit understanding tasks, substantially improving evaluation validity and diagnostic precision.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Understanding clients' thoughts and beliefs is fundamental in counseling, yet current evaluations of LLM therapists often fail to assess this ability. Existing evaluation methods rely on client simulators that clearly disclose internal states to the therapist, making it difficult to determine whether an LLM therapist can uncover unexpressed perspectives. To address this limitation, we introduce MindVoyager, a novel evaluation framework featuring a controllable and realistic client simulator which dynamically adapts itself based on the ongoing counseling session, offering a more realistic and challenging evaluation environment. We further introduce evaluation metrics that assess the exploration ability of LLM therapists by measuring their thorough understanding of client's beliefs and thoughts.

Problem

• Assessing LLM therapists' ability to uncover unexpressed client perspectives
• Developing realistic client simulator for dynamic counseling evaluation
• Measuring LLM therapists' thorough understanding of client beliefs

Innovation

• MindVoyager framework for realistic client simulation
• Dynamic adaptation based on counseling session context
• Metrics measuring LLM therapist exploration ability