Feature-Based Dual Visual Feature Extraction Model for Compound Multimodal Emotion Recognition

2025-03-21

AI Summary

Composite emotion recognition in real-world scenarios faces significant challenges from modality conflicts and heightened uncertainty, particularly under complex audiovisual cues. To address this, we propose a dual-visual-backbone collaborative architecture that, for the first time, performs feature-level fusion of complementary visual representations extracted by Vision Transformers (ViT) and ResNet, jointly with audio features, enabling end-to-end multimodal modeling. This design enhances visual representation diversity and robustness while effectively mitigating inter-modal inconsistency. Extensive experiments on the C-EXPR-DB and MELD benchmarks demonstrate that our method substantially outperforms single-backbone baselines in composite emotion recognition—especially exhibiting superior generalization on noisy and low-quality audiovisual samples. The source code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This article presents our results for the eighth Affective Behavior Analysis in-the-wild (ABAW) competition.Multimodal emotion recognition (ER) has important applications in affective computing and human-computer interaction. However, in the real world, compound emotion recognition faces greater issues of uncertainty and modal conflicts. For the Compound Expression (CE) Recognition Challenge,this paper proposes a multimodal emotion recognition method that fuses the features of Vision Transformer (ViT) and Residual Network (ResNet). We conducted experiments on the C-EXPR-DB and MELD datasets. The results show that in scenarios with complex visual and audio cues (such as C-EXPR-DB), the model that fuses the features of ViT and ResNet exhibits superior performance.Our code are avalible on https://github.com/MyGitHub-ax/8th_ABAW

Problem

• Addresses uncertainty and modal conflicts in compound emotion recognition
• Proposes ViT-ResNet fusion for multimodal emotion recognition
• Improves performance in complex visual-audio scenarios like C-EXPR-DB

Innovation

• Fuses Vision Transformer and ResNet features
• Targets compound emotion recognition challenges
• Improves performance in complex multimodal scenarios