Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild

2026-06-22

AI Summary

This work addresses the challenges of 4D reconstruction of dynamic non-rigid objects from in-the-wild monocular videos, where severe occlusions, large deformations, and scarce training data hinder performance. The authors propose a test-time optimization framework that achieves temporally consistent single-view 3D predictions through causal latent conditioning. It integrates a deformable 3D Gaussian splatting representation with an occlusion-aware optimization scheme and leverages a viewpoint-conditioned diffusion prior to reconstruct unobserved regions. By uniquely combining temporally coherent 3D estimation with occlusion-aware diffusion prior optimization, the method significantly enhances the robustness, completeness, and geometric fidelity of 4D reconstructions, outperforming existing approaches on challenging real-world sequences featuring extreme non-rigid motion and heavy occlusion.

Abstract

Reconstructing dynamic non-rigid objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well. We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then ``sculpt'' this representation to match the input video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.

Problem

• 4D reconstruction
• non-rigid objects
• monocular video
• occlusions
• in-the-wild

Innovation

• 4D reconstruction
• single-view 3D estimation
• Gaussian Splatting
• diffusion prior
• occlusion-aware optimization