An End-to-End Learning-Based Multi-Sensor Fusion for Autonomous Vehicle Localization

2025-03-07

AI Summary

To address the limitations of conventional multi-sensor fusion for autonomous vehicle localization—namely, its reliance on Gaussian assumptions, manual hyperparameter tuning, and poor generalization to long-tail scenarios—this paper proposes an end-to-end learnable fusion framework that jointly regresses ego-vehicle pose directly from raw sensor inputs, eliminating explicit uncertainty modeling and hand-crafted priors. Our key contributions are: (1) a high-order feature-based sensor encoder; (2) a customized end-to-end fusion network; and (3) a jointly optimized pose regression architecture that implicitly encodes feature-level uncertainty. Evaluated on real-world road scenes, our method achieves an 18.7% improvement in localization accuracy and demonstrates significantly enhanced robustness compared to both classical filters (EKF, UKF) and state-of-the-art learning-based approaches. Code and experimental validation video are available at https://youtu.be/q4iuobMbjME.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multi-sensor fusion is essential for autonomous vehicle localization, as it is capable of integrating data from various sources for enhanced accuracy and reliability. The accuracy of the integrated location and orientation depends on the precision of the uncertainty modeling. Traditional methods of uncertainty modeling typically assume a Gaussian distribution and involve manual heuristic parameter tuning. However, these methods struggle to scale effectively and address long-tail scenarios. To address these challenges, we propose a learning-based method that encodes sensor information using higher-order neural network features, thereby eliminating the need for uncertainty estimation. This method significantly eliminates the need for parameter fine-tuning by developing an end-to-end neural network that is specifically designed for multi-sensor fusion. In our experiments, we demonstrate the effectiveness of our approach in real-world autonomous driving scenarios. Results show that the proposed method outperforms existing multi-sensor fusion methods in terms of both accuracy and robustness. A video of the results can be viewed at https://youtu.be/q4iuobMbjME.

Problem

• Improves autonomous vehicle localization accuracy
• Eliminates manual parameter tuning in sensor fusion
• Addresses long-tail scenarios in uncertainty modeling

Innovation

• End-to-end neural network for sensor fusion
• Higher-order neural network encodes sensor data
• Eliminates manual parameter tuning and uncertainty estimation