EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety

2025-04-13

AI Summary

LLM-driven AI agents interacting with psychologically vulnerable users pose significant mental health risks. Method: We propose EmoAgent, a multi-agent framework for risk assessment and mitigation, featuring a novel virtual user simulation mechanism that integrates clinical psychometric scales (PHQ-9, PDI, PANSS), real-time affective state monitoring, dynamic risk evaluation, and LLM-based safety interventions for proactive risk identification and correction. Contribution/Results: Empirical evaluation on mainstream role-playing chatbots reveals that emotionally expressive dialogue exacerbates psychological distress in over 34.4% of vulnerable users; integrating the EmoGuard module reduces deterioration rates significantly and substantially improves interaction safety. This work pioneers the deep integration of standardized clinical psychological assessment into AI interaction safety architectures, establishing a verifiable, deployable risk governance paradigm for LLM applications in mental-health-sensitive contexts.

Technology Category

• Natural Language Processing: Safety and Robustness
• Humans and AI: Emotional Intelligence
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent

Problem

• Assessing mental health risks in human-AI interactions
• Mitigating psychological harm for vulnerable users
• Monitoring and improving AI chatbot safety

Innovation

• Multi-agent AI framework for mental health safety
• Simulates vulnerable users with clinical assessment tools
• Monitors and mitigates psychological risks in real-time