Minimizing Occlusion Effect on Multi-View Camera Perception in BEV with Multi-Sensor Fusion

2025-01-10

AI Summary

Autonomous driving perception—particularly BEV vehicle segmentation and detection—suffers severe performance degradation under occlusions caused by rain, fog, and dust. This work first systematically quantifies the spatial distribution patterns of such occlusions in BEV space using the nuScenes dataset. We propose a cross-modal occlusion compensation mechanism grounded in BEV feature projection modeling and multi-sensor spatiotemporal alignment, enabling tightly coupled fusion of LiDAR, radar, and camera modalities. Additionally, we design an occlusion-aware adaptive weighting network to dynamically suppress unreliable visual features. Experiments demonstrate a 12.7% improvement in vehicle segmentation mIoU and a 91.3% BEV detection recall rate under heavy rain/fog conditions, significantly enhancing perception robustness in adverse weather.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Autonomous driving technology is rapidly evolving, offering the potential for safer and more efficient transportation. However, the performance of these systems can be significantly compromised by the occlusion on sensors due to environmental factors like dirt, dust, rain, and fog. These occlusions severely affect vision-based tasks such as object detection, vehicle segmentation, and lane recognition. In this paper, we investigate the impact of various kinds of occlusions on camera sensor by projecting their effects from multi-view camera images of the nuScenes dataset into the Bird's-Eye View (BEV) domain. This approach allows us to analyze how occlusions spatially distribute and influence vehicle segmentation accuracy within the BEV domain. Despite significant advances in sensor technology and multi-sensor fusion, a gap remains in the existing literature regarding the specific effects of camera occlusions on BEV-based perception systems. To address this gap, we use a multi-sensor fusion technique that integrates LiDAR and radar sensor data to mitigate the performance degradation caused by occluded cameras. Our findings demonstrate that this approach significantly enhances the accuracy and robustness of vehicle segmentation tasks, leading to more reliable autonomous driving systems.

Problem

• Autonomous Driving
• Object Recognition
• Environmental Obstructions

Innovation

• LiDAR-Radar Fusion
• Obstacle Recognition
• Autonomous Driving