Learnable Infinite Taylor Gaussian for Dynamic View Rendering

2024-12-05

arXiv.org

AI Summary

Modeling temporal evolution of 3D Gaussian ellipsoids in dynamic scenes remains challenging due to inherent difficulties in capturing nonlinear spatiotemporal dynamics and poor convergence under sparse photometric supervision. To address this, we propose an explicit dynamic parameterization method based on learnable infinite-order Taylor series, jointly modeling time-varying position, rotation, and scale in a unified, differentiable framework. This is the first work to embed infinite-order Taylor expansion into an end-to-end differentiable pipeline, harmonizing the expressiveness of neural implicit representations with the physical interpretability of polynomial dynamics—without requiring auxiliary motion supervision. By integrating temporal encoding into Gaussian parameterization and coupling it with dynamic 3D Gaussian rendering, our approach achieves state-of-the-art performance on multiple dynamic novel-view synthesis benchmarks. It significantly improves motion boundary sharpness and inter-frame consistency while accelerating inference by 3.2× compared to neural ODE-based methods.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Capturing the temporal evolution of Gaussian properties such as position, rotation, and scale is a challenging task due to the vast number of time-varying parameters and the limited photometric data available, which generally results in convergence issues, making it difficult to find an optimal solution. While feeding all inputs into an end-to-end neural network can effectively model complex temporal dynamics, this approach lacks explicit supervision and struggles to generate high-quality transformation fields. On the other hand, using time-conditioned polynomial functions to model Gaussian trajectories and orientations provides a more explicit and interpretable solution, but requires significant handcrafted effort and lacks generalizability across diverse scenes. To overcome these limitations, this paper introduces a novel approach based on a learnable infinite Taylor Formula to model the temporal evolution of Gaussians. This method offers both the flexibility of an implicit network-based approach and the interpretability of explicit polynomial functions, allowing for more robust and generalizable modeling of Gaussian dynamics across various dynamic scenes. Extensive experiments on dynamic novel view rendering tasks are conducted on public datasets, demonstrating that the proposed method achieves state-of-the-art performance in this domain. More information is available on our project page(https://ellisonking.github.io/TaylorGaussian).

Problem

• Modeling temporal evolution of Gaussian properties with limited data
• Balancing implicit network flexibility and explicit polynomial interpretability
• Achieving robust dynamic view rendering across diverse scenes

Innovation

• Learnable infinite Taylor Formula for Gaussians
• Combines implicit network flexibility with interpretability
• State-of-the-art dynamic view rendering performance