Integrated Multimodal Sensing and Communication: Challenges, Technologies, and Architectures

2025-06-25

AI Summary

To address the weak environmental characterization capability and poor scalability of conventional unimodal Integrated Sensing and Communication (ISAC) systems in 6G, this paper proposes a novel multimodal collaborative ISAC paradigm. It introduces three heterogeneous architectural variants—F-MAC (Federated MAC), I-MAC (Integrated MAC), and R-MAC (Reconfigurable MAC)—to enable cross-modal data fusion and distributed multi-agent collaboration. The approach innovatively integrates large language model (LLM)-driven semantic communication with multi-agent coordination mechanisms, overcoming key bottlenecks including multi-source heterogeneous data alignment, excessive communication overhead, and dynamic architectural adaptation. Experimental evaluation demonstrates that the F-MAC-based prototype system achieves approximately 80% higher sensing accuracy than traditional unimodal ISAC schemes, significantly improving sensing completeness and system resilience under complex, dynamic scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning
• Multiagent Systems: Agent Communication

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The evolution towards 6G networks requires the intelligent integration of communication and sensing capabilities to support diverse and complex applications, such as autonomous driving and immersive services. However, existing integrated sensing and communication (ISAC) systems predominantly rely on single-modal sensors as primary participants, which leads to a limited representation of environmental features and significant performance bottlenecks under the emerging requirements of 6G applications. This limitation motivates a paradigm shift from single-modal to multimodal ISAC. In this article, we first analyze the key challenges in realizing multimodal ISAC, including the fusion of heterogeneous multimodal data, the high communication overhead among distributed sensors, and the design of efficient and scalable system architectures. We then introduce several enabling technologies, such as large AI models, semantic communication, and multi-agent systems, that hold promise for addressing these challenges. To operationalize these technologies, we zoom into three architectural paradigms: fusion-based multimodal ISAC (F-MAC), interaction-based multimodal ISAC (I-MAC), and relay-based multimodal ISAC (R-MAC), each tailored to organize devices and modalities for efficient collaboration in different scenarios. Thereafter, a case study is presented based on the F-MAC scheme, demonstrating that the scheme achieves more comprehensive sensing and improves sensing accuracy by approximately 80% compared to conventional single-modal ISAC systems. Finally, we discuss several open issues to be addressed in the future.

Problem

• Integrating multimodal sensing and communication for 6G networks
• Overcoming single-modal limitations in ISAC systems
• Addressing data fusion and scalability in multimodal ISAC

Innovation

• Multimodal ISAC with AI and semantic communication
• Fusion-based, interaction-based, relay-based architectures
• 80% accuracy boost in sensing performance