Navigating the Emotion Tree: Hierarchical Hyperbolic RAG for Multimodal Emotion Recognition

2026-05-16

AI Summary

This work addresses the limitations of existing multimodal emotion recognition approaches, which often overlook the hierarchical psychological structure of emotion categories and lack external knowledge, rendering them susceptible to noise and hindering fine-grained classification performance. To overcome these challenges, the authors propose HyperEmo-RAG, a novel framework that integrates a structured emotion knowledge base into multimodal emotion recognition for the first time. It models the hierarchical emotion taxonomy via hyperbolic embeddings in Poincaré balls and introduces an evidence-graph-based knowledge injection mechanism to preserve graph topological information. By synergistically combining retrieval-augmented generation (RAG), Tree-Aware Attention, and an EmotionGraphFormer module, the framework enables coarse-to-fine emotion reasoning. Extensive experiments demonstrate that HyperEmo-RAG significantly outperforms state-of-the-art methods across multiple benchmark datasets, achieving enhanced accuracy and robustness in fine-grained emotion recognition.

Abstract

Multimodal emotion recognition aims to integrate text, audio, and video sources to understand human affective states. Although multimodal large language models excel at multimodal reasoning, they typically treat emotion categories as independent labels, ignoring the rich hierarchical taxonomy of human psychology. Moreover, lacking external contextual knowledge makes them highly susceptible to over-interpreting noisy cues, further complicating fine-grained emotion classification. To address these issues, we propose \textbf{HyperEmo-RAG}, a retrieval-augmented generation framework that leverages a structured emotional knowledge base. Our framework introduces two key innovations. 1) Hierarchical hyperbolic grounding. Recognizing the inherent hierarchical tree structure of emotion taxonomies, we jointly embed hierarchical emotion labels and multimodal samples into a continuous hyperbolic space (Poincaré ball) and design a hierarchical beam-search deliberation process that progressively retrieves samples from coarse to fine-grained levels. 2) Structured evidence injection. Based on the retrieved evidence, we construct an evidence graph and inject the structured knowledge as explicit cognitive context into the LLM through a Tree-Aware Attention mechanism and an EmotionGraphFormer, preserving the integrity of graph-structured information. Experiments on multiple datasets demonstrate that HyperEmo-RAG significantly outperforms existing methods.

Problem

• multimodal emotion recognition
• emotion taxonomy
• hierarchical structure
• fine-grained emotion classification
• contextual knowledge

Innovation

• Hierarchical hyperbolic embedding
• Retrieval-augmented generation
• Multimodal emotion recognition
• Structured knowledge injection
• Emotion taxonomy