Gaussian Splatting Lucas-Kanade

2024-07-16

AI Summary

This work addresses the failure of existing dynamic Gaussian splatting methods in scenarios with minimal camera motion but highly dynamic scene objects. We propose an analytical motion modeling framework that requires no semantic or geometric priors. Our core innovation is the first analytical integration of the Lucas-Kanade optical flow method into dynamic Gaussian splatting, enabling derivation of an analytical velocity field for the forward deformation network. By performing temporal integration, we precisely estimate scene flow, jointly constraining both the 2D pixel-level motion and 3D spatial positions of Gaussian ellipsoids. This approach eliminates distributional bias inherent in data-driven priors and explicitly models generalized motion dynamics. Evaluated on real and synthetic scenes featuring low camera motion and high object dynamics, our method significantly improves geometric accuracy and motion consistency. It establishes a new differentiable, analytical, and prior-free motion modeling paradigm for dynamic NeRF-style reconstruction.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Motion and Path Planning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Gaussian Splatting and its dynamic extensions are effective for reconstructing 3D scenes from 2D images when there is significant camera movement to facilitate motion parallax and when scene objects remain relatively static. However, in many real-world scenarios, these conditions are not met. As a consequence, data-driven semantic and geometric priors have been favored as regularizers, despite their bias toward training data and their neglect of broader movement dynamics. Departing from this practice, we propose a novel analytical approach that adapts the classical Lucas-Kanade method to dynamic Gaussian splatting. By leveraging the intrinsic properties of the forward warp field network, we derive an analytical velocity field that, through time integration, facilitates accurate scene flow computation. This enables the precise enforcement of motion constraints on warp fields, thus constraining both 2D motion and 3D positions of the Gaussians. Our method excels in reconstructing highly dynamic scenes with minimal camera movement, as demonstrated through experiments on both synthetic and real-world scenes.

Problem

• Reconstructing 3D scenes from 2D images with minimal camera movement
• Overcoming limitations of static scene assumptions in dynamic environments
• Enforcing accurate motion constraints on 3D Gaussian warp fields

Innovation

• Adapts Lucas-Kanade to dynamic Gaussian splatting
• Derives analytical velocity field for scene flow
• Enforces motion constraints on warp fields precisely