Multi-Modal Sensor Fusion for Proactive Blockage Prediction in mmWave Vehicular Networks

2025-07-21

AI Summary

To address signal blockage caused by dynamic obstacles (e.g., vehicles, pedestrians, infrastructure) in millimeter-wave vehicular networks, this paper proposes a multimodal perception–based proactive occlusion prediction framework for infrastructure-to-vehicle (I2V) communication. The framework fuses camera, LiDAR, millimeter-wave radar, and GPS data using modality-specific deep learning models and employs a Softmax-weighted fusion strategy guided by validation performance to enable efficient heterogeneous sensor collaboration. Its key contribution is enabling up to 1.5-second lookahead occlusion prediction. Experimental results demonstrate that the camera-only model achieves an F1-score of 97.1% with 89.8 ms inference latency; integrating camera and radar further improves the F1-score to 97.2%, delivering high accuracy, low latency, and strong robustness under dynamic urban scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Vehicular communication systems operating in the millimeter wave (mmWave) band are highly susceptible to signal blockage from dynamic obstacles such as vehicles, pedestrians, and infrastructure. To address this challenge, we propose a proactive blockage prediction framework that utilizes multi-modal sensing, including camera, GPS, LiDAR, and radar inputs in an infrastructure-to-vehicle (I2V) setting. This approach uses modality-specific deep learning models to process each sensor stream independently and fuses their outputs using a softmax-weighted ensemble strategy based on validation performance. Our evaluations, for up to 1.5s in advance, show that the camera-only model achieves the best standalone trade-off with an F1-score of 97.1% and an inference time of 89.8ms. A camera+radar configuration further improves accuracy to 97.2% F1 at 95.7ms. Our results display the effectiveness and efficiency of multi-modal sensing for mmWave blockage prediction and provide a pathway for proactive wireless communication in dynamic environments.

Problem

• Predict mmWave signal blockage in vehicular networks
• Fuse multi-modal sensor data for accurate prediction
• Improve proactive wireless communication in dynamic environments

Innovation

• Multi-modal sensor fusion for blockage prediction
• Deep learning models for independent sensor processing
• Softmax-weighted ensemble strategy for fusion