Multi-Modal Sensing and Fusion in mmWave Beamforming for Connected Vehicles: A Transformer Based Framework

2026-02-14

AI Summary

This work addresses the high overhead of millimeter-wave beam training and low communication efficiency in highly dynamic vehicular networks by proposing a novel beam prediction framework that integrates multimodal perception with a Transformer architecture. The approach uniquely combines modality-specific encoders to extract features from diverse sensory inputs and employs a multi-head cross-modal attention mechanism to model inter-modal dependencies, enabling effective information fusion and accurate beam prediction. Evaluated in real-world 60 GHz vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) scenarios, the method achieves a top-15 prediction accuracy of 96.72% with an average power loss of only 0.77 dB. Furthermore, it reduces beam search space and latency overhead by 76.56% and 86.81%, respectively, substantially enhancing both the efficiency and accuracy of initial link establishment.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Millimeter wave (mmWave) communication, utilizing beamforming techniques to address the inherent path loss limitation, is considered as one of the key technologies to support ever increasing high throughput and low latency demands of connected vehicles. However, adopting standard defined beamforming approach in highly dynamic vehicular environments often incurs high beam training overheads and reduction in the available airtime for communications, which is mainly due to exchanging pilot signals and exhaustive beam measurements. To this end, we present a multi-modal sensing and fusion learning framework as a potential alternative solution to reduce such overheads. In this framework, we first extract the representative features from the sensing modalities by modality specific encoders, then, utilize multi-head cross-modal attention to learn dependencies and correlations between different modalities, and subsequently fuse the multimodal features to obtain predicted top-k beams so that the best line-of-sight links can be proactively established. To show the generalizability of the proposed framework, we perform a comprehensive experiment in four different vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) scenarios from real world multimodal and 60 GHz mmWave wireless sensing data. The experiment reveals that the proposed framework (i) achieves up to 96.72% accuracy on predicting top-15 beams correctly, (ii) incurs roughly 0.77 dB average power loss, and (iii) improves the overall latency and beam searching space overheads by 86.81% and 76.56% respectively for top-15 beams compared to standard defined approach.

Problem

• mmWave beamforming
• beam training overhead
• connected vehicles
• dynamic vehicular environments
• pilot signaling

Innovation

• multi-modal sensing
• transformer-based fusion
• mmWave beamforming
• cross-modal attention
• connected vehicles