Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey

2025-03-16

AI Summary

Current multimodal chain-of-thought (MCoT) reasoning lacks a systematic survey, suffering from conceptual ambiguity, methodological fragmentation, and incomplete modality coverage. This work establishes the first unified taxonomy and comprehensive classification framework for MCoT, encompassing six modalities: image, video, speech, 3D, structured data, and cross-modal combinations. We rigorously formalize foundational definitions and synthesize core methodologies—including cross-modal alignment, stepwise multimodal reasoning modeling, interpretability analysis, and task-driven evaluation. Furthermore, we identify critical pathways toward multimodal artificial general intelligence (AGI) and articulate key open challenges. The survey critically analyzes over 200 seminal works, offering a methodological guide and technology roadmap for high-impact domains such as robotics, healthcare, and autonomous driving. To our knowledge, this is the first authoritative, comprehensive survey dedicated to MCoT.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

By extending the advantage of chain-of-thought (CoT) reasoning in human-like step-by-step processes to multimodal contexts, multimodal CoT (MCoT) reasoning has recently garnered significant research attention, especially in the integration with multimodal large language models (MLLMs). Existing MCoT studies design various methodologies and innovative reasoning paradigms to address the unique challenges of image, video, speech, audio, 3D, and structured data across different modalities, achieving extensive success in applications such as robotics, healthcare, autonomous driving, and multimodal generation. However, MCoT still presents distinct challenges and opportunities that require further focus to ensure consistent thriving in this field, where, unfortunately, an up-to-date review of this domain is lacking. To bridge this gap, we present the first systematic survey of MCoT reasoning, elucidating the relevant foundational concepts and definitions. We offer a comprehensive taxonomy and an in-depth analysis of current methodologies from diverse perspectives across various application scenarios. Furthermore, we provide insights into existing challenges and future research directions, aiming to foster innovation toward multimodal AGI.

Problem

• Extends chain-of-thought reasoning to multimodal contexts.
• Addresses challenges in integrating image, video, speech, and 3D data.
• Provides a systematic survey and future directions for MCoT research.

Innovation

• Extends CoT reasoning to multimodal contexts
• Integrates with multimodal large language models
• Addresses challenges across diverse data modalities