SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes

2024-10-22

arXiv.org

AI Summary

Photometrically inconsistent reconstruction in dynamic specular scenes remains challenging due to inaccurate specular reflectance modeling and difficulty adapting to time-varying illumination. Method: We propose the first photometrically consistent 3D reconstruction framework for glossy dynamic scenes, integrating 3D Gaussian splatting with physics-based rendering (PBR). A learnable deformation field models surface dynamics; a novel residual normal correction module enhances dynamic normal estimation accuracy; and a deformable environment map captures time-varying lighting. A coarse-to-fine training strategy jointly optimizes all components. Contribution/Results: Our method achieves, for the first time, photorealistic view synthesis on real-world dynamic glossy scenes. Quantitative and qualitative evaluations demonstrate significant improvements over existing state-of-the-art methods on complex specular dynamic sequences, particularly in preserving temporal coherence, specular fidelity, and geometric detail under varying illumination.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. Previous methods extending 3DGS to model dynamic scenes have struggled to represent specular surfaces accurately. Our method addresses this limitation by introducing a residual correction technique for accurate surface normal computation during deformation, complemented by a deformable environment map that adapts to time-varying lighting conditions. We implement a coarse-to-fine training strategy that significantly enhances scene geometry and specular color prediction. It is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes. See our project page for video results at https://cdfan0627.github.io/spectromotion/.

Problem

• Accurate 3D reconstruction of dynamic specular scenes
• Overcoming limitations in representing specular surfaces with 3DGS
• Enhancing geometry and color prediction in varying lighting conditions

Innovation

• Combines 3DGS with PBR and deformation fields
• Uses residual correction for accurate normals
• Implements coarse-to-fine training strategy