Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Tools

2024-08-03

AI Summary

To address the lack of standardization, low automation, and poor inter-rater agreement with human evaluation in safety assessment of mental health chatbots, this study proposes and validates a novel multimodal safety evaluation framework. The framework comprises a 100-item benchmark test set, an expert-consensus response guideline, and an agentic evaluation architecture integrating LLM-based scoring, semantic embedding comparison, and real-time knowledge retrieval. It introduces, for the first time, an expert-collaborative validation mechanism and a dynamic evaluation paradigm grounded in real-time data retrieval. Experimental results demonstrate that the framework significantly enhances response safety and reliability. Crucially, the agentic evaluation achieves a Cohen’s kappa of κ = 0.89 against human annotations—the highest reported to date—thereby empirically validating the critical role of real-time information integration in trustworthy safety assessment.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements

Abstract

Objective: This study aims to develop and validate an evaluation framework to ensure the safety and reliability of mental health chatbots, which are increasingly popular due to their accessibility, human-like interactions, and context-aware support. Materials and Methods: We created an evaluation framework with 100 benchmark questions and ideal responses, and five guideline questions for chatbot responses. This framework, validated by mental health experts, was tested on a GPT-3.5-turbo-based chatbot. Automated evaluation methods explored included large language model (LLM)-based scoring, an agentic approach using real-time data, and embedding models to compare chatbot responses against ground truth standards. Results: The results highlight the importance of guidelines and ground truth for improving LLM evaluation accuracy. The agentic method, dynamically accessing reliable information, demonstrated the best alignment with human assessments. Adherence to a standardized, expert-validated framework significantly enhanced chatbot response safety and reliability. Discussion: Our findings emphasize the need for comprehensive, expert-tailored safety evaluation metrics for mental health chatbots. While LLMs have significant potential, careful implementation is necessary to mitigate risks. The superior performance of the agentic approach underscores the importance of real-time data access in enhancing chatbot reliability. Conclusion: The study validated an evaluation framework for mental health chatbots, proving its effectiveness in improving safety and reliability. Future work should extend evaluations to accuracy, bias, empathy, and privacy to ensure holistic assessment and responsible integration into healthcare. Standardized evaluations will build trust among users and professionals, facilitating broader adoption and improved mental health support through technology.

Problem

• Develop evaluation framework for mental health chatbot safety
• Validate framework using expert guidelines and LLM-based tools
• Enhance chatbot reliability through real-time data access

Innovation

• Developed LLM-based scoring for chatbot evaluation
• Implemented agentic approach using real-time data
• Used embedding models to compare responses