CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis

2024-07-09

arXiv.org

AI Summary

Existing methods assume strong inter-modal correlations, limiting their performance on weakly correlated multimodal data. To address this, we propose CorMulT, a two-stage semi-supervised model: in the pretraining stage, modality-correlation-aware contrastive learning dynamically captures fine-grained associations among text, image, and audio; in the prediction stage, the learned cross-modal correlation coefficients are adaptively integrated into modality-specific representations to guide sentiment classification. This is the first work to explicitly model and leverage learnable correlation coefficients for enhancing multimodal sentiment analysis. CorMulT synergistically integrates multimodal Transformers, contrastive learning, and semi-supervised learning. On CMU-MOSEI, it significantly outperforms state-of-the-art methods, achieving substantial robustness gains—particularly on samples with low inter-modal correlation.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal sentiment analysis is an active research area that combines multiple data modalities, e.g., text, image and audio, to analyze human emotions and benefits a variety of applications. Existing multimodal sentiment analysis methods can be classified as modality interaction-based methods, modality transformation-based methods and modality similarity-based methods. However, most of these methods highly rely on the strong correlations between modalities, and cannot fully uncover and utilize the correlations between modalities to enhance sentiment analysis. Therefore, these methods usually achieve bad performance for identifying the sentiment of multimodal data with weak correlations. To address this issue, we proposed a two-stage semi-supervised model termed Correlation-aware Multimodal Transformer (CorMulT) which consists pre-training stage and prediction stage. At the pre-training stage, a modality correlation contrastive learning module is designed to efficiently learn modality correlation coefficients between different modalities. At the prediction stage, the learned correlation coefficients are fused with modality representations to make the sentiment prediction. According to the experiments on the popular multimodal dataset CMU-MOSEI, CorMulT obviously surpasses state-of-the-art multimodal sentiment analysis methods.

Problem

• Enhance sentiment analysis by uncovering multimodal correlations
• Address weak correlation challenges in multimodal data
• Improve performance over existing sentiment analysis methods

Innovation

• Semi-supervised multimodal Transformer for sentiment analysis
• Modality correlation contrastive learning in pre-training
• Fusion of correlation coefficients for prediction