Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video

2025-01-24

AI Summary

To address the severe degradation in 3D reconstruction and ego-motion estimation caused by sensor noise, motion blur, and temporal asynchrony in real-world videos, this paper introduces Robust-Ego3D—the first noise-robust 3D vision benchmark—and establishes a scalable pipeline for synthesizing realistic noisy data. Methodologically, we propose CorrGS: a Gaussian Splatting–based framework that leverages RGB-D–guided test-time adaptive optimization. By jointly modeling inter-frame visual correspondences and enforcing geometric-appearance alignment, CorrGS dynamically corrects noisy observations, enabling end-to-end mapping from noisy video to clean, high-fidelity 3D reconstructions. Evaluated on both synthetic and real-world noisy sequences, CorrGS reduces ego-motion estimation error by 37% and improves 3D reconstruction PSNR by 5.2 dB over state-of-the-art methods—particularly excelling under fast motion and dynamic lighting conditions.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: State Estimation
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We aim to redefine robust ego-motion estimation and photorealistic 3D reconstruction by addressing a critical limitation: the reliance on noise-free data in existing models. While such sanitized conditions simplify evaluation, they fail to capture the unpredictable, noisy complexities of real-world environments. Dynamic motion, sensor imperfections, and synchronization perturbations lead to sharp performance declines when these models are deployed in practice, revealing an urgent need for frameworks that embrace and excel under real-world noise. To bridge this gap, we tackle three core challenges: scalable data generation, comprehensive benchmarking, and model robustness enhancement. First, we introduce a scalable noisy data synthesis pipeline that generates diverse datasets simulating complex motion, sensor imperfections, and synchronization errors. Second, we leverage this pipeline to create Robust-Ego3D, a benchmark rigorously designed to expose noise-induced performance degradation, highlighting the limitations of current learning-based methods in ego-motion accuracy and 3D reconstruction quality. Third, we propose Correspondence-guided Gaussian Splatting (CorrGS), a novel test-time adaptation method that progressively refines an internal clean 3D representation by aligning noisy observations with rendered RGB-D frames from clean 3D map, enhancing geometric alignment and appearance restoration through visual correspondence. Extensive experiments on synthetic and real-world data demonstrate that CorrGS consistently outperforms prior state-of-the-art methods, particularly in scenarios involving rapid motion and dynamic illumination.

Problem

• 3D Reconstruction
• Self-Motion Estimation
• Noise Robustness

Innovation

• CorrGS
• Robust-Ego3D
• Noise-resilient 3D reconstruction