StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams

2025-06-10

AI Summary

This paper addresses real-time online dynamic 3D reconstruction from uncalibrated video streams, proposing the first fully feed-forward framework capable of streaming arbitrary-length videos. Methodologically: (1) a static encoder coupled with a probabilistic sampling mechanism enhances robustness in localizing 3D Gaussian splatting (3DGS) primitives; (2) a bidirectional deformation field-based dynamic decoder jointly models geometry and motion under local temporal observations, balancing reconstruction accuracy and long-term stability. Contributions include: the first real-time dynamic 3DGS generation from uncalibrated videos; no reliance on global optimization or historical frame buffering; state-of-the-art performance on both static and dynamic benchmarks; and support for infinite-length video streaming, significantly improving reconstruction quality and computational efficiency.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Real-time reconstruction of dynamic 3D scenes from uncalibrated video streams is crucial for numerous real-world applications. However, existing methods struggle to jointly address three key challenges: 1) processing uncalibrated inputs in real time, 2) accurately modeling dynamic scene evolution, and 3) maintaining long-term stability and computational efficiency. To this end, we introduce StreamSplat, the first fully feed-forward framework that transforms uncalibrated video streams of arbitrary length into dynamic 3D Gaussian Splatting (3DGS) representations in an online manner, capable of recovering scene dynamics from temporally local observations. We propose two key technical innovations: a probabilistic sampling mechanism in the static encoder for 3DGS position prediction, and a bidirectional deformation field in the dynamic decoder that enables robust and efficient dynamic modeling. Extensive experiments on static and dynamic benchmarks demonstrate that StreamSplat consistently outperforms prior works in both reconstruction quality and dynamic scene modeling, while uniquely supporting online reconstruction of arbitrarily long video streams. Code and models are available at https://github.com/nickwzk/StreamSplat.

Problem

• Real-time 3D reconstruction from uncalibrated videos
• Accurate modeling of dynamic scene evolution
• Maintaining long-term stability and computational efficiency

Innovation

• Feed-forward framework for uncalibrated video streams
• Probabilistic sampling in static 3DGS encoder
• Bidirectional deformation field for dynamic modeling