A re-calibration method for object detection with multi-modal alignment bias in autonomous driving

2024-05-27

arXiv.org

AI Summary

In autonomous driving, minor calibration errors between LiDAR and camera sensors—e.g., ±0.5° rotation or ±5 cm translation—severely degrade multimodal 3D detection performance, yet existing methods lack robust modeling of calibration drift. To address this, we propose a semantic segmentation-guided online recalibration method: (1) a differentiable affine transformation module estimates and compensates cross-modal geometric misalignments in an end-to-end manner, supervised by image-level semantic segmentation; (2) a multimodal feature alignment loss jointly optimizes detection and recalibration; and (3) the method is seamlessly integrated into mainstream detectors such as EPNet++. Evaluated on nuScenes under synthetic calibration perturbations, our approach improves mAP by 12.3% while significantly enhancing robustness—without introducing any inference-time overhead.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Multi-modal object detection in autonomous driving has achieved great breakthroughs due to the usage of fusing complementary information from different sensors. The calibration in fusion between sensors such as LiDAR and camera is always supposed to be precise in previous work. However, in reality, calibration matrices are fixed when the vehicles leave the factory, but vibration, bumps, and data lags may cause calibration bias. As the research on the calibration influence on fusion detection performance is relatively few, flexible calibration dependency multi-sensor detection method has always been attractive. In this paper, we conducted experiments on SOTA detection method EPNet++ and proved slight bias on calibration can reduce the performance seriously. We also proposed a re-calibration model based on semantic segmentation which can be combined with a detection algorithm to improve the performance and robustness of multi-modal calibration bias.

Problem

• Addresses multi-modal alignment bias in autonomous driving object detection
• Explores impact of calibration errors on sensor fusion performance
• Proposes re-calibration model to improve detection robustness

Innovation

• Recalibration model for multi-modal alignment bias
• Semantic segmentation based calibration correction
• Improves detection robustness against calibration errors