Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities

2025-08-08

AI Summary

Existing multimodal sentiment recognition methods overlook the heterogeneous reconstruction difficulty caused by missing modalities, leading to suboptimal robustness. Method: This paper proposes a hardness-aware dynamic curriculum learning framework. It introduces a novel multi-view hardness evaluation mechanism that jointly models *direct hardness* (reconstruction error) and *indirect hardness* (cross-modal mutual information), and designs a retrieval-based dynamic curriculum strategy for adaptive, hierarchical sample training. Contribution/Results: Compared with static or uniform training paradigms, our framework significantly improves model robustness on highly challenging missing-modality samples. Extensive experiments on standard benchmarks—including CMU-MOSEI and IEMOCAP—demonstrate consistent performance gains, achieving an average accuracy improvement of 2.3% over current state-of-the-art methods. These results validate both the effectiveness and generalizability of hardness-aware curriculum learning for sentiment recognition under modality missingness.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Missing modalities have recently emerged as a critical research direction in multimodal emotion recognition (MER). Conventional approaches typically address this issue through missing modality reconstruction. However, these methods fail to account for variations in reconstruction difficulty across different samples, consequently limiting the model's ability to handle hard samples effectively. To overcome this limitation, we propose a novel Hardness-Aware Dynamic Curriculum Learning framework, termed HARDY-MER. Our framework operates in two key stages: first, it estimates the hardness level of each sample, and second, it strategically emphasizes hard samples during training to enhance model performance on these challenging instances. Specifically, we first introduce a Multi-view Hardness Evaluation mechanism that quantifies reconstruction difficulty by considering both Direct Hardness (modality reconstruction errors) and Indirect Hardness (cross-modal mutual information). Meanwhile, we introduce a Retrieval-based Dynamic Curriculum Learning strategy that dynamically adjusts the training curriculum by retrieving samples with similar semantic information and balancing the learning focus between easy and hard instances. Extensive experiments on benchmark datasets demonstrate that HARDY-MER consistently outperforms existing methods in missing-modality scenarios. Our code will be made publicly available at https://github.com/HARDY-MER/HARDY-MER.

Problem

• Addresses missing modalities in multimodal emotion recognition
• Improves handling of hard samples via dynamic curriculum learning
• Enhances reconstruction difficulty estimation with multi-view evaluation

Innovation

• Hardness-Aware Dynamic Curriculum Learning framework
• Multi-view Hardness Evaluation mechanism
• Retrieval-based Dynamic Curriculum Learning strategy