ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimation

2025-06-05

AI Summary

Existing methods for joint estimation of physical parameters—geometry, appearance, state, and material—under sparse multi-view video settings suffer from either dense-view dependency or error accumulation due to sequential optimization. This work proposes a parameter-sensitivity-guided progressive joint optimization framework, enabling the first end-to-end differentiable 4D physics-aware reconstruction from sparse views. Our approach unifies neural radiance fields (NeRF), differentiable physics simulation, sensitivity-driven parameter scheduling, and implicit 4D representation to overcome two fundamental bottlenecks: infeasibility of full-parameter joint optimization and error propagation in cascaded pipelines. Evaluated on PAC-NeRF and Spring-Gaus benchmarks, our method achieves state-of-the-art performance in future state prediction, novel-view synthesis, and material estimation. The resulting reconstructions exhibit strong physical consistency, facilitating robust digital twin construction.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: State Estimation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Neural rendering has made significant strides in 3D reconstruction and novel view synthesis. With the integration with physics, it opens up new applications. The inverse problem of estimating physics from visual data, however, still remains challenging, limiting its effectiveness for applications like physically accurate digital twin creation in robotics and XR. Existing methods that incorporate physics into neural rendering frameworks typically require dense multi-view videos as input, making them impractical for scalable, real-world use. When presented with sparse multi-view videos, the sequential optimization strategy used by existing approaches introduces significant error accumulation, e.g., poor initial 3D reconstruction leads to bad material parameter estimation in subsequent stages. Instead of sequential optimization, directly optimizing all parameters at the same time also fails due to the highly non-convex and often non-differentiable nature of the problem. We propose ProJo4D, a progressive joint optimization framework that gradually increases the set of jointly optimized parameters guided by their sensitivity, leading to fully joint optimization over geometry, appearance, physical state, and material property. Evaluations on PAC-NeRF and Spring-Gaus datasets show that ProJo4D outperforms prior work in 4D future state prediction, novel view rendering of future state, and material parameter estimation, demonstrating its effectiveness in physically grounded 4D scene understanding. For demos, please visit the project webpage: https://daniel03c1.github.io/ProJo4D/

Problem

• Estimating physics from sparse multi-view videos
• Overcoming error accumulation in sequential optimization
• Jointly optimizing geometry, appearance, physics, and materials

Innovation

• Progressive joint optimization framework
• Gradually increases optimized parameters
• Jointly optimizes geometry, appearance, physics