4D Scaffold Gaussian Splatting for Memory Efficient Dynamic Scene Reconstruction

2024-11-26

arXiv.org

AI Summary

To address the trade-off between excessive memory consumption and low dynamic-region fidelity in 4D Gaussian reconstruction for dynamic scenes, this paper proposes an efficient representation framework based on sparse 4D grid anchors. Methodologically, it introduces: (1) a learnable 4D scaffolding structure that models neural 4D Gaussians within local spatiotemporal neighborhoods of anchors; (2) a temporal coverage-aware anchor adaptation strategy for precise dynamic-region coverage and redundancy reduction; and (3) temporally varying opacity derived jointly from a neural velocity field and generalized Gaussians, enabling accurate modeling of non-rigid deformation and occlusion. Integrated with gradient-weighted optimization and compressed feature vector encoding, the method achieves real-time rendering and high-fidelity reconstruction while reducing memory overhead by 97.8% compared to 4DGS—setting a new state-of-the-art in visual quality.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Existing 4D Gaussian methods for dynamic scene reconstruction offer high visual fidelity and fast rendering. However, these methods suffer from excessive memory and storage demands, which limits their practical deployment. This paper proposes a 4D anchor-based framework that retains visual quality and rendering speed of 4D Gaussians while significantly reducing storage costs. Our method extends 3D scaffolding to 4D space, and leverages sparse 4D grid-aligned anchors with compressed feature vectors. Each anchor models a set of neural 4D Gaussians, each of which represent a local spatiotemporal region. In addition, we introduce a temporal coverage-aware anchor growing strategy to effectively assign additional anchors to under-reconstructed dynamic regions. Our method adjusts the accumulated gradients based on Gaussians' temporal coverage, improving reconstruction quality in dynamic regions. To reduce the number of anchors, we further present enhanced formulations of neural 4D Gaussians. These include the neural velocity, and the temporal opacity derived from a generalized Gaussian distribution. Experimental results demonstrate that our method achieves state-of-the-art visual quality and 97.8% storage reduction over 4DGS.

Problem

• Reducing storage overhead in 4D dynamic scene reconstruction
• Maintaining high-fidelity rendering in dynamic regions
• Optimizing anchor allocation for under-reconstructed areas

Innovation

• 4D anchor-based framework for dynamic scene modeling
• MLP-processed anchors spawn compact neural 4D Gaussians
• Dynamic-aware anchor growing improves reconstruction quality