Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos

2024-12-26

AI Summary

Reconstructing dynamic 3D scenes from unsynchronized, uncalibrated multi-view videos remains challenging due to the absence of temporal alignment and intrinsic/extrinsic camera parameters. Method: We propose a weakly supervised learning framework leveraging human motion as a natural spatiotemporal anchor. For the first time, monocular 3D human pose estimates (SMPL) serve as weak supervision to jointly optimize camera poses, inter-frame temporal offsets, and dynamic Neural Radiance Fields (NeRFs). To address severe non-convexity, we introduce a multi-resolution voxel grid and a robust progressive optimization strategy. Results: Our method achieves high-fidelity spatiotemporal calibration and dynamic scene reconstruction on real-world videos—without time synchronization or prior camera calibration. Experiments demonstrate significant improvements in robustness and geometric-appearance fidelity under complex, unconstrained scenarios.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent works on dynamic neural field reconstruction assume input from synchronized multi-view videos with known poses. These input constraints are often unmet in real-world setups, making the approach impractical. We demonstrate that unsynchronized videos with unknown poses can generate dynamic neural fields if the videos capture human motion. Humans are one of the most common dynamic subjects whose poses can be estimated using state-of-the-art methods. While noisy, the estimated human shape and pose parameters provide a decent initialization for the highly non-convex and under-constrained problem of training a consistent dynamic neural representation. Given the sequences of pose and shape of humans, we estimate the time offsets between videos, followed by camera pose estimations by analyzing 3D joint locations. Then, we train dynamic NeRF employing multiresolution rids while simultaneously refining both time offsets and camera poses. The setup still involves optimizing many parameters, therefore, we introduce a robust progressive learning strategy to stabilize the process. Experiments show that our approach achieves accurate spatiotemporal calibration and high-quality scene reconstruction in challenging conditions.

Problem

• Multi-view Video Synchronization
• Unknown Human Pose Estimation
• High-quality Dynamic 3D Scene Reconstruction

Innovation

• Asynchronous Video Reconstruction
• Dynamic Neural Radiance Fields
• Temporal Spatial Calibration