Time of the Flight of the Gaussians: Optimizing Depth Indirectly in Dynamic Radiance Fields

2025-05-08

AI Summary

Monocular continuous-wave time-of-flight (C-ToF) cameras face fundamental challenges in direct depth estimation, while dynamic scene reconstruction suffers from phase ambiguity and motion blur. To address these issues, we propose the first real-time dynamic 3D reconstruction framework that jointly leverages implicit depth optimization and 3D Gaussian Splatting. Our method establishes a physically grounded forward model of raw C-ToF measurements, incorporates temporal geometric regularization, and introduces two robustness-oriented heuristics to mitigate phase ambiguities and motion artifacts. Experiments demonstrate high-fidelity geometric reconstruction on fast-moving scenes (e.g., baseball bat swings), achieving depth accuracy comparable to or exceeding neural radiance rendering approaches. Crucially, our method operates at millisecond latency with a 100× speedup in inference time over prior methods—marking the first demonstration of real-time, high-accuracy dynamic 3D reconstruction driven solely by monocular C-ToF input.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We present a method to reconstruct dynamic scenes from monocular continuous-wave time-of-flight (C-ToF) cameras using raw sensor samples that achieves similar or better accuracy than neural volumetric approaches and is 100x faster. Quickly achieving high-fidelity dynamic 3D reconstruction from a single viewpoint is a significant challenge in computer vision. In C-ToF radiance field reconstruction, the property of interest-depth-is not directly measured, causing an additional challenge. This problem has a large and underappreciated impact upon the optimization when using a fast primitive-based scene representation like 3D Gaussian splatting, which is commonly used with multi-view data to produce satisfactory results and is brittle in its optimization otherwise. We incorporate two heuristics into the optimization to improve the accuracy of scene geometry represented by Gaussians. Experimental results show that our approach produces accurate reconstructions under constrained C-ToF sensing conditions, including for fast motions like swinging baseball bats. https://visual.cs.brown.edu/gftorf

Problem

• Optimizing depth indirectly in dynamic radiance fields
• Reconstructing dynamic scenes from monocular C-ToF cameras
• Improving accuracy of scene geometry with Gaussian representations

Innovation

• Uses C-ToF cameras for dynamic scene reconstruction
• Applies heuristics to optimize Gaussian-based geometry
• Achieves 100x faster than neural volumetric methods