Sensing and Understanding the World over Air: A Large Multimodal Model for Mobile Networks

2025-11-17

AI Summary

Current wireless networks lack dedicated multimodal large language models (WMLMs), and general-purpose models fail to effectively capture the physical semantics of wireless signals. Method: This paper introduces the first wireless-native multimodal large model, anchoring its architecture on wireless signals as the primary modality and adopting a GPT-style design. It employs cross-modal contrastive learning and joint multimodal modeling trained on large-scale real-world wireless data. Contribution/Results: The work pioneers the validation of wireless signals as a viable universal modality within large models, establishing the “wireless-native” paradigm. Experiments demonstrate that the proposed model significantly outperforms existing small-scale models and general-purpose multimodal LLMs on key tasks—including channel estimation, target sensing, and network optimization—exhibiting superior physical-world perception and understanding capabilities. This model serves as a foundational architecture for intelligent 6G networks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Large models (LMs), such as ChatGPT, have made a significant impact across diverse domains and hold great potential to facilitate the evolution of network intelligence. Wireless-native multi-modal large models (WMLMs) can sense and understand the physical world through multi-modal data, serving as a key enabler that integrates communication, sensing, and intelligence, and thus they can boost various smart services to billions of users. However, research on WMLMs remains in its infancy, and the construction of domain-specific multi-modal large models for wireless networks is still underexplored. In this paper, we outlines the key characteristics of WMLMs and summarizes existing methods, on the basis of which a wireless-native multimodal training paradigm is proposed. Specifically, we constructed a GPT-style WMLM model and trained it on a real-world large-scale dataset, leveraging wireless signals as an anchor modality for contrastive learning. Our approach demonstrates outstanding performance compared with existing small-scale models and large multi-modal models, validating the feasibility of using wireless signals as a universal modality and highlighting WMLM's potential to emerge as a new paradigm for future wireless networks.

Problem

• Developing wireless-native multimodal large models for network intelligence
• Integrating communication, sensing, and intelligence using wireless signals
• Addressing the underexplored construction of domain-specific multimodal models

Innovation

• Wireless-native multimodal training paradigm for networks
• GPT-style model using wireless signals as anchor modality
• Contrastive learning on real-world large-scale dataset