Efficient 4D Gaussian Stream with Low Rank Adaptation

2025-02-23

AI Summary

To address the inherent trade-off among high computational cost, excessive bandwidth demand, and degraded rendering quality in long-duration dynamic scene novel view synthesis, this paper proposes the first 4D Gaussian-based dynamic reconstruction framework tailored for streaming video. Methodologically, we innovatively integrate Low-Rank Adaptation (LoRA) into 3D Gaussian Splatting to construct an incrementally updatable low-rank deformation model; further, we introduce block-wise spatiotemporal alignment and online optimization to enable continual learning and frame-level incremental reconstruction over arbitrarily long video sequences. Compared to state-of-the-art offline methods, our framework achieves comparable rendering fidelity (PSNR/SSIM), while reducing streaming bandwidth by 90% and substantially lowering GPU memory consumption and computational overhead. This provides a scalable solution for real-time, high-fidelity, long-duration dynamic novel view synthesis.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Time-Series/Data Streams
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Recent methods have made significant progress in synthesizing novel views with long video sequences. This paper proposes a highly scalable method for dynamic novel view synthesis with continual learning. We leverage the 3D Gaussians to represent the scene and a low-rank adaptation-based deformation model to capture the dynamic scene changes. Our method continuously reconstructs the dynamics with chunks of video frames, reduces the streaming bandwidth by $90%$ while maintaining high rendering quality comparable to the off-line SOTA methods.

Problem

• Dynamic novel view synthesis
• Low-rank adaptation deformation
• Reduced streaming bandwidth

Innovation

• 4D Gaussian Stream technique
• Low Rank Adaptation model
• Dynamic scene reconstruction efficiency