EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World
🤖 AI Summary
Current evaluation paradigms for empathetic support dialogues over-rely on generalized empathy, failing to capture how well responses align with users’ unique psychological traits and contextual needs. To address this limitation, this work proposes EmoHarbor—an automated evaluation framework grounded in a “user-as-judge” paradigm. EmoHarbor employs a psychology-informed multi-agent architecture (Chain-of-Agent) to simulate the user’s internal cognitive and affective processes, decomposing personalized assessment into three specialized roles. The framework introduces a comprehensive benchmark comprising 100 real-user personas across 10 distinct dimensions. Experimental results reveal that while state-of-the-art large language models exhibit general empathic capabilities, they consistently lack adaptability to individual contextual nuances, thereby underscoring the critical need—and a promising new direction—for research into personalized emotional support.
Application Category
📝 Abstract
Current evaluation paradigms for emotional support conversations tend to reward generic empathetic responses, yet they fail to assess whether the support is genuinely personalized to users'unique psychological profiles and contextual needs. We introduce EmoHarbor, an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. EmoHarbor employs a Chain-of-Agent architecture that decomposes users'internal processes into three specialized roles, enabling agents to interact with supporters and complete assessments in a manner similar to human users. We instantiate this benchmark using 100 real-world user profiles that cover a diverse range of personality traits and situations, and define 10 evaluation dimensions of personalized support quality. Comprehensive evaluation of 20 advanced LLMs on EmoHarbor reveals a critical insight: while these models excel at generating empathetic responses, they consistently fail to tailor support to individual user contexts. This finding reframes the central challenge, shifting research focus from merely enhancing generic empathy to developing truly user-aware emotional support. EmoHarbor provides a reproducible and scalable framework to guide the development and evaluation of more nuanced and user-aware emotional support systems.
Problem
Research questions and friction points this paper is trying to address.
emotional support
personalization
evaluation paradigm
user-awareness
empathy
Innovation
Methods, ideas, or system contributions that make the work stand out.
EmoHarbor
personalized emotional support
Chain-of-Agent
User-as-a-Judge
empathy evaluation
🔎 Similar Papers
No similar papers found.
