ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion

2025-07-07

AI Summary

In multimodal emotion and intent recognition, sensor failures or data corruption often cause missing modalities, yet existing methods suffer from excessive inter-modal coupling and generation distortion during feature reconstruction. To address this, we propose an Attention-driven Diffusion Model (ADM) that decouples modality-specific representation learning via independently trained modality encoders and guides the diffusion process using cross-modal attention to achieve high-fidelity, distribution-consistent imputation of missing modality features. The framework enables end-to-end joint optimization and cross-modal enhancement, simultaneously improving robustness under both complete and partial-observability scenarios. Evaluated on the IEMOCAP and MIntRec benchmarks, ADM outperforms current state-of-the-art methods, achieving average accuracy improvements of 3.2% for emotion recognition and 4.1% for intent recognition.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Multimodal emotion and intent recognition is essential for automated human-computer interaction, It aims to analyze users' speech, text, and visual information to predict their emotions or intent. One of the significant challenges is that missing modalities due to sensor malfunctions or incomplete data. Traditional methods that attempt to reconstruct missing information often suffer from over-coupling and imprecise generation processes, leading to suboptimal outcomes. To address these issues, we introduce an Attention-based Diffusion model for Missing Modalities feature Completion (ADMC). Our framework independently trains feature extraction networks for each modality, preserving their unique characteristics and avoiding over-coupling. The Attention-based Diffusion Network (ADN) generates missing modality features that closely align with authentic multimodal distribution, enhancing performance across all missing-modality scenarios. Moreover, ADN's cross-modal generation offers improved recognition even in full-modality contexts. Our approach achieves state-of-the-art results on the IEMOCAP and MIntRec benchmarks, demonstrating its effectiveness in both missing and complete modality scenarios.

Problem

• Addresses missing modalities in multimodal emotion recognition
• Overcomes over-coupling in traditional feature completion methods
• Enhances recognition accuracy in partial and full modality scenarios

Innovation

• Attention-based Diffusion model for missing modalities
• Independent feature extraction networks per modality
• Cross-modal generation enhances recognition accuracy