SENSE-7: Taxonomy and Dataset for Measuring User Perceptions of Empathy in Sustained Human-AI Conversations

2025-09-19

AI Summary

Traditional digital empathy research predominantly models internal emotional states, overlooking users’ subjective, contextualized, and relational perceptions of empathy in sustained human–AI dialogue. Method: We propose a human-centered taxonomy of empathic behaviors and introduce Sense-7—the first fine-grained, real-world workplace dialogue dataset comprising 672 anonymized multi-turn conversations, with turn-level empathy annotations, user demographic/psychographic features, and rich contextual metadata. Our framework uniquely integrates individualized, contextual, and relational dimensions of empathy perception. Contribution/Results: Leveraging Sense-7, we train an LLM-based empathy recognizer achieving Spearman’s ρ = 0.369 and 48.7% accuracy on a five-point empathy classification task—demonstrating the high personalization and context-dependency of empathy perception. This work establishes an empirical foundation and methodological framework for designing adaptive, user-aware empathic AI systems.

Technology Category

• Humans and AI: Emotional Intelligence
• Natural Language Processing: Conversational AI/Dialog Systems
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Empathy is increasingly recognized as a key factor in human-AI communication, yet conventional approaches to "digital empathy" often focus on simulating internal, human-like emotional states while overlooking the inherently subjective, contextual, and relational facets of empathy as perceived by users. In this work, we propose a human-centered taxonomy that emphasizes observable empathic behaviors and introduce a new dataset, Sense-7, of real-world conversations between information workers and Large Language Models (LLMs), which includes per-turn empathy annotations directly from the users, along with user characteristics, and contextual details, offering a more user-grounded representation of empathy. Analysis of 695 conversations from 109 participants reveals that empathy judgments are highly individualized, context-sensitive, and vulnerable to disruption when conversational continuity fails or user expectations go unmet. To promote further research, we provide a subset of 672 anonymized conversation and provide exploratory classification analysis, showing that an LLM-based classifier can recognize 5 levels of empathy with an encouraging average Spearman $ρ$=0.369 and Accuracy=0.487 over this set. Overall, our findings underscore the need for AI designs that dynamically tailor empathic behaviors to user contexts and goals, offering a roadmap for future research and practical development of socially attuned, human-centered artificial agents.

Problem

• Measuring subjective user perceptions of empathy in AI conversations
• Addressing limitations in current digital empathy approaches and datasets
• Developing AI that adapts empathic behaviors to user context

Innovation

• Human-centered taxonomy focusing on observable empathic behaviors
• Introducing Sense-7 dataset with real-world user-annotated conversations
• LLM-based classifier recognizing five empathy levels from interactions