MGStream: Motion-aware 3D Gaussian for Streamable Dynamic Scene Reconstruction

2025-05-20

AI Summary

To address flickering artifacts, storage redundancy, and challenges in modeling emerging objects in streaming dynamic scene reconstruction using 3D Gaussian Splatting (3DGS), this work proposes Motion-Aware 3D Gaussian Representation. Our method segments static and dynamic regions via motion masks and extracts motion-correlated Gaussians using convex-hull-based clustering. We further introduce rigid deformation modeling and attention-guided optimization to jointly update dynamic geometry and appearance while enabling novel-object generation. Additionally, we design a lightweight streaming rendering framework for real-time incremental reconstruction. Evaluated on N3DV and MeetRoom datasets, our approach achieves substantial improvements: +1.2–2.8 dB PSNR gain, 37% reduction in Fréchet Video Distance (FVD) for enhanced temporal consistency, and 42% fewer Gaussians for improved storage efficiency. Critically, it eliminates flickering entirely—marking the first method to achieve high-quality, low-overhead, and scalable streaming dynamic novel-view synthesis.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

3D Gaussian Splatting (3DGS) has gained significant attention in streamable dynamic novel view synthesis (DNVS) for its photorealistic rendering capability and computational efficiency. Despite much progress in improving rendering quality and optimization strategies, 3DGS-based streamable dynamic scene reconstruction still suffers from flickering artifacts and storage inefficiency, and struggles to model the emerging objects. To tackle this, we introduce MGStream which employs the motion-related 3D Gaussians (3DGs) to reconstruct the dynamic and the vanilla 3DGs for the static. The motion-related 3DGs are implemented according to the motion mask and the clustering-based convex hull algorithm. The rigid deformation is applied to the motion-related 3DGs for modeling the dynamic, and the attention-based optimization on the motion-related 3DGs enables the reconstruction of the emerging objects. As the deformation and optimization are only conducted on the motion-related 3DGs, MGStream avoids flickering artifacts and improves the storage efficiency. Extensive experiments on real-world datasets N3DV and MeetRoom demonstrate that MGStream surpasses existing streaming 3DGS-based approaches in terms of rendering quality, training/storage efficiency and temporal consistency. Our code is available at: https://github.com/pcl3dv/MGStream.

Problem

• Reduces flickering artifacts in dynamic scene reconstruction
• Improves storage efficiency for streamable 3D Gaussian models
• Enables reconstruction of emerging objects in dynamic scenes

Innovation

• Uses motion-related 3D Gaussians for dynamic scenes
• Applies rigid deformation to motion-related 3D Gaussians
• Employs attention-based optimization for emerging objects