4DSTR: Advancing Generative 4D Gaussians with Spatial-Temporal Rectification for High-Quality and Consistent 4D Generation

2025-11-10

AI Summary

Existing 4D generation methods suffer from limited spatiotemporal consistency and poor modeling of rapid motion, primarily due to the lack of effective joint spatiotemporal representations. To address this, we propose 4DSTR—a generative network based on 4D Gaussian lattices. Our method introduces two key innovations: (1) a spatiotemporal correction mechanism that explicitly optimizes Gaussian ellipsoid scaling and rotation deformations via time-correlation modeling, ensuring temporal coherence; and (2) an adaptive spatial densification and dynamic pruning strategy that responds in real time to geometric changes induced by abrupt motion. By integrating differentiable 4D rendering with learnable Gaussian point insertion and removal, 4DSTR enables end-to-end video-to-4D generation. Evaluated on standard benchmarks, 4DSTR achieves significant improvements in reconstruction accuracy, spatiotemporal continuity, and robustness to fast motion—setting new state-of-the-art performance.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Motion & Tracking
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Remarkable advances in recent 2D image and 3D shape generation have induced a significant focus on dynamic 4D content generation. However, previous 4D generation methods commonly struggle to maintain spatial-temporal consistency and adapt poorly to rapid temporal variations, due to the lack of effective spatial-temporal modeling. To address these problems, we propose a novel 4D generation network called 4DSTR, which modulates generative 4D Gaussian Splatting with spatial-temporal rectification. Specifically, temporal correlation across generated 4D sequences is designed to rectify deformable scales and rotations and guarantee temporal consistency. Furthermore, an adaptive spatial densification and pruning strategy is proposed to address significant temporal variations by dynamically adding or deleting Gaussian points with the awareness of their pre-frame movements. Extensive experiments demonstrate that our 4DSTR achieves state-of-the-art performance in video-to-4D generation, excelling in reconstruction quality, spatial-temporal consistency, and adaptation to rapid temporal movements.

Problem

• Maintaining spatial-temporal consistency in 4D generation
• Addressing poor adaptation to rapid temporal variations
• Enhancing reconstruction quality in dynamic 4D content

Innovation

• Modulates generative 4D Gaussian Splatting with rectification
• Rectifies deformable scales and rotations for consistency
• Dynamically adds or deletes Gaussian points adaptively