Towards Unified Multimodal Financial Forecasting: Integrating Sentiment Embeddings and Market Indicators via Cross-Modal Attention

2025-08-18

AI Summary

To address the representational fragmentation arising from isolated modeling of market time-series data and news text in stock prediction, this paper proposes STONK—a unified multimodal financial forecasting framework. STONK jointly encodes numerical market indicators (e.g., price, volume) and sentiment-aware news embeddings via feature concatenation and cross-modal attention, enabling dynamic inter-modal interaction. Unlike unimodal baselines, it effectively captures synergistic effects between market behavior and sentiment-driven signals. Backtesting on real-world stock market data demonstrates that STONK achieves significantly higher daily directional accuracy than purely numerical baselines. Ablation studies confirm the necessity of each component, while multi-strategy integration analysis validates its robustness, generalizability, and scalability. Overall, STONK establishes a reproducible empirical paradigm for deep fusion of heterogeneous, multi-source financial signals.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We propose STONK (Stock Optimization using News Knowledge), a multimodal framework integrating numerical market indicators with sentiment-enriched news embeddings to improve daily stock-movement prediction. By combining numerical & textual embeddings via feature concatenation and cross-modal attention, our unified pipeline addresses limitations of isolated analyses. Backtesting shows STONK outperforms numeric-only baselines. A comprehensive evaluation of fusion strategies and model configurations offers evidence-based guidance for scalable multimodal financial forecasting. Source code is available on GitHub

Problem

• Improving daily stock movement prediction accuracy
• Integrating numerical market indicators with sentiment embeddings
• Addressing limitations of isolated financial analysis methods

Innovation

• Integrating sentiment embeddings with market indicators
• Using cross-modal attention for unified pipeline
• Evaluating fusion strategies for scalable forecasting