Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles

2025-06-26

AI Summary

To address the insufficient robustness of single-sensor perception for autonomous vehicles under adverse weather and complex urban conditions, this paper systematically investigates multimodal sensor fusion, unifying data-level, feature-level, and decision-level fusion paradigms within a coherent formalism. We propose a deep learning–based cross-modal alignment and representation learning framework that, for the first time, integrates vision-language models (VLMs) and large language models (LLMs) into the sensor fusion pipeline—thereby enhancing adaptability and uncertainty modeling in end-to-end autonomous driving. We establish a comprehensive evaluation framework across major benchmarks—including nuScenes, BDD100K, and Oxford Radar RobotCar—and demonstrate significant improvements in object detection and semantic segmentation accuracy under challenging conditions such as rain, fog, and nighttime.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multi-sensor fusion plays a critical role in enhancing perception for autonomous driving, overcoming individual sensor limitations, and enabling comprehensive environmental understanding. This paper first formalizes multi-sensor fusion strategies into data-level, feature-level, and decision-level categories and then provides a systematic review of deep learning-based methods corresponding to each strategy. We present key multi-modal datasets and discuss their applicability in addressing real-world challenges, particularly in adverse weather conditions and complex urban environments. Additionally, we explore emerging trends, including the integration of Vision-Language Models (VLMs), Large Language Models (LLMs), and the role of sensor fusion in end-to-end autonomous driving, highlighting its potential to enhance system adaptability and robustness. Our work offers valuable insights into current methods and future directions for multi-sensor fusion in autonomous driving.

Problem

• Enhancing perception for autonomous driving via multi-sensor fusion.
• Reviewing deep learning-based fusion methods for environmental understanding.
• Exploring sensor fusion's role in adverse weather and urban scenarios.

Innovation

• Multi-sensor fusion enhances autonomous driving perception
• Deep learning methods formalize fusion strategies
• Integrates VLMs and LLMs for system robustness