MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

2025-10-17

AI Summary

To address three key challenges in multi-camera Structure-from-Motion (SfM) reconstruction for autonomous driving—unreliable pose estimation, abundant outliers in road surface point clouds, and low optimization efficiency—this paper proposes MRASfM, an integrated framework. Methodologically: (1) a multi-camera rigid-body constraint model is introduced to improve initial pose accuracy; (2) plane-prior-guided road point cloud filtering and joint optimization significantly suppress outliers; and (3) a cross-camera collaborative Bundle Adjustment coupled with a scene association–assembly module enables coarse-to-fine cascaded aggregation. Evaluated on the nuScenes dataset, MRASfM achieves an absolute pose error of 0.124 m, demonstrating robustness and state-of-the-art performance across large-scale, complex real-world driving scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Structure from Motion (SfM) estimates camera poses and reconstructs point clouds, forming a foundation for various tasks. However, applying SfM to driving scenes captured by multi-camera systems presents significant difficulties, including unreliable pose estimation, excessive outliers in road surface reconstruction, and low reconstruction efficiency. To address these limitations, we propose a Multi-camera Reconstruction and Aggregation Structure-from-Motion (MRASfM) framework specifically designed for driving scenes. MRASfM enhances the reliability of camera pose estimation by leveraging the fixed spatial relationships within the multi-camera system during the registration process. To improve the quality of road surface reconstruction, our framework employs a plane model to effectively remove erroneous points from the triangulated road surface. Moreover, treating the multi-camera set as a single unit in Bundle Adjustment (BA) helps reduce optimization variables to boost efficiency. In addition, MRASfM achieves multi-scene aggregation through scene association and assembly modules in a coarse-to-fine fashion. We deployed multi-camera systems on actual vehicles to validate the generalizability of MRASfM across various scenes and its robustness in challenging conditions through real-world applications. Furthermore, large-scale validation results on public datasets show the state-of-the-art performance of MRASfM, achieving 0.124 absolute pose error on the nuScenes dataset.

Problem

• Enhances camera pose estimation reliability in multi-camera driving systems
• Improves road surface reconstruction quality by removing erroneous points
• Boosts reconstruction efficiency through unified bundle adjustment optimization

Innovation

• Leverages fixed multi-camera spatial relationships for registration
• Uses plane model to remove erroneous road surface points
• Treats multi-camera set as single unit in Bundle Adjustment