Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges

2025-05-15

arXiv.org

AI Summary

This paper addresses the challenges of integrating and deeply understanding multimodal financial data—including textual, audio, visual, and video content, alongside market, macroeconomic, and alternative data. To this end, it proposes a novel paradigm: the Multimodal Financial Foundation Model (MFFM). Methodologically, it introduces the first systematic conceptual framework for MFFMs, transcending the linguistic limitations of existing Financial Large Language Models (FinLLMs) by unifying cross-modal alignment, heterogeneous representation fusion, and finance-domain adaptive pretraining and fine-tuning. Key contributions include: (1) establishing a comprehensive development roadmap for MFFMs; (2) open-sourcing the Awesome-MFFMs repository—a curated collection of models, datasets, and benchmarks; and (3) enabling complex financial applications such as intelligent investment research, risk management, and regulatory oversight, thereby advancing the field from unimodal analysis to cross-modal deep reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Financial Large Language Models (FinLLMs), such as open FinGPT and proprietary BloombergGPT, have demonstrated great potential in select areas of financial services. Beyond this earlier language-centric approach, Multimodal Financial Foundation Models (MFFMs) can digest interleaved multimodal financial data, including fundamental data, market data, data analytics, macroeconomic, and alternative data (e.g., natural language, audio, images, and video). In this position paper, presented at the MFFM Workshop joined with ACM International Conference on AI in Finance (ICAIF) 2024, we describe the progress, prospects, and challenges of MFFMs. This paper also highlights ongoing research on FinAgents in the extbf{SecureFinAI Lab}footnote{https://openfin.engineering.columbia.edu/} at Columbia University. We believe that MFFMs will enable a deeper understanding of the underlying complexity associated with numerous financial tasks and data, streamlining the operation of financial services and investment processes. Github Repo https://github.com/Open-Finance-Lab/Awesome-MFFMs/.

Problem

• Develop multimodal models for financial data analysis
• Enhance understanding of complex financial tasks
• Streamline financial services and investment processes

Innovation

• Multimodal models process diverse financial data
• Integrates natural language, audio, images, video
• Enhances financial task understanding and operations