Multi-Modal Opinion Integration for Financial Sentiment Analysis using Cross-Modal Attention

2025-12-03

AI Summary

To address the challenges of fusing heterogeneous opinion modalities and insufficient fine-grained cross-modal interaction in financial sentiment analysis, this paper proposes an end-to-end multimodal sentiment analysis framework. We introduce a Financial Multimodal Head Cross-Attention (FMHCA) mechanism, specifically designed for financial scenarios, to enable dynamic feature interaction between timeliness- and popularity-oriented textual modalities. Additionally, we propose a multimodal decomposition bilinear pooling module to enhance fusion efficiency. Textual representations are extracted using Chinese-wwm-ext BERT, augmented with additional Transformer layers for refined semantic modeling. Evaluated on a large-scale financial sentiment dataset covering 837 companies, our model achieves 83.5% accuracy—outperforming the BERT+Transformer baseline by 21 percentage points—and demonstrates significantly improved capability in fine-grained emotion recognition.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

In recent years, financial sentiment analysis of public opinion has become increasingly important for market forecasting and risk assessment. However, existing methods often struggle to effectively integrate diverse opinion modalities and capture fine-grained interactions across them. This paper proposes an end-to-end deep learning framework that integrates two distinct modalities of financial opinions: recency modality (timely opinions) and popularity modality (trending opinions), through a novel cross-modal attention mechanism specifically designed for financial sentiment analysis. While both modalities consist of textual data, they represent fundamentally different information channels: recency-driven market updates versus popularity-driven collective sentiment. Our model first uses BERT (Chinese-wwm-ext) for feature embedding and then employs our proposed Financial Multi-Head Cross-Attention (FMHCA) structure to facilitate information exchange between these distinct opinion modalities. The processed features are optimized through a transformer layer and fused using multimodal factored bilinear pooling for classification into negative, neutral, and positive sentiment. Extensive experiments on a comprehensive dataset covering 837 companies demonstrate that our approach achieves an accuracy of 83.5%, significantly outperforming baselines including BERT+Transformer by 21 percent. These results highlight the potential of our framework to support more accurate financial decision-making and risk management.

Problem

• Integrates recency and popularity opinion modalities for sentiment analysis
• Uses cross-modal attention to capture fine-grained interactions between modalities
• Improves financial sentiment classification accuracy for market forecasting

Innovation

• Cross-modal attention mechanism for financial sentiment analysis
• BERT-based feature embedding with Financial Multi-Head Cross-Attention
• Multimodal factored bilinear pooling for sentiment classification