Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies

2026-07-01

AI Summary

This study addresses the significant comprehension gap faced by current large language models in zero-shot fine-grained emotion recognition. For the first time, it conducts a unified, example-free evaluation of three leading commercial models—Claude, ChatGPT, and Gemini—in real-world API settings across 13 fine-grained emotion categories. Leveraging zero-shot prompt engineering, stratified sampling, and evaluation metrics including macro F1-score and accuracy—supplemented by McNemar’s significance test—the analysis reveals that Gemini achieves the best performance (accuracy: 39.9%, macro-F1: 0.363). However, all models struggle notably with recognizing emotions such as “love,” “confusion,” and “shame,” and no statistically significant differences emerge among them overall, highlighting a shared bottleneck and performance ceiling in zero-shot affective understanding.

Abstract

Emotion recognition in natural language is a foundational challenge in affective computing, with critical implications for human-computer interaction, mental health support, and conversational AI. This paper presents a rigorous, unified zero-shot evaluation of three leading commercial large language models: Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash). The models were queried through their respective production APIs as of April 2026 on a fine-grained 13-class emotion classification task. Using a stratified 1,000-sentence sample from the boltuix/emotions dataset, which comprises 131,306 sentences across 13 categories, a single uniform prompt with no exemplars was applied identically across all models. Gemini achieves the highest accuracy (39.9%) and macro-F1 score (0.363), followed by GPT-5.4 (38.8%, macro-F1 = 0.291) and Claude (38.0%, macro-F1 = 0.159). All models excel on sarcasm and desire while consistently failing on love, confusion, and shame. McNemar tests reveal no statistically significant pairwise differences (p > 0.10), suggesting convergence at a shared zero-shot ceiling. Claude's markedly lower macro-F1 score exposes a class-imbalance prediction bias. These findings highlight the current limitations of frontier AI systems in zero-shot fine-grained emotion classification.

Problem

• emotion recognition
• zero-shot evaluation
• fine-grained emotion
• large language models
• affective computing

Innovation

• zero-shot evaluation
• fine-grained emotion classification
• large language models
• affective computing
• class-imbalance bias