A Trustworthy Method for Multimodal Emotion Recognition

2025-08-11

AI Summary

Existing multimodal emotion recognition methods primarily focus on improving accuracy while neglecting prediction reliability, rendering them vulnerable to noise, corrupted inputs, and out-of-distribution data. To address this, we propose TER (Trustworthy Emotion Recognition), the first framework to incorporate uncertainty estimation into multimodal emotion recognition. TER employs confidence-weighted fusion of modality-specific outputs and introduces a novel evaluation suite—including trustworthy accuracy, trustworthy recall, and trustworthy F1—to quantify predictive reliability. It jointly optimizes uncertainty quantification, robust multimodal fusion, and adaptive threshold-based decision-making in an end-to-end manner. Evaluated on IEMOCAP and Music-video datasets, TER achieves trustworthy F1 scores of 0.7511 and 0.9035, respectively, alongside an overall accuracy of 82.40%, consistently outperforming state-of-the-art methods. This demonstrates substantial improvements in model reliability and generalization robustness under distributional shifts and input perturbations.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Existing emotion recognition methods mainly focus on enhancing performance by employing complex deep models, typically resulting in significantly higher model complexity. Although effective, it is also crucial to ensure the reliability of the final decision, especially for noisy, corrupted and out-of-distribution data. To this end, we propose a novel emotion recognition method called trusted emotion recognition (TER), which utilizes uncertainty estimation to calculate the confidence value of predictions. TER combines the results from multiple modalities based on their confidence values to output the trusted predictions. We also provide a new evaluation criterion to assess the reliability of predictions. Specifically, we incorporate trusted precision and trusted recall to determine the trusted threshold and formulate the trusted Acc. and trusted F1 score to evaluate the model's trusted performance. The proposed framework combines the confidence module that accordingly endows the model with reliability and robustness against possible noise or corruption. The extensive experimental results validate the effectiveness of our proposed model. The TER achieves state-of-the-art performance on the Music-video, achieving 82.40% Acc. In terms of trusted performance, TER outperforms other methods on the IEMOCAP and Music-video, achieving trusted F1 scores of 0.7511 and 0.9035, respectively.

Problem

• Ensuring reliable emotion recognition with noisy data
• Reducing model complexity in multimodal emotion recognition
• Evaluating prediction reliability using trusted metrics

Innovation

• Uncertainty estimation for prediction confidence
• Multimodal fusion based on confidence values
• New trusted metrics for reliability evaluation