Problem
Research questions and friction points this paper is trying to address.
monocular 3D face reconstruction
3D Morphable Model
real-time performance
dense priors
optimization efficiency
RealDenseFace: Real-time Monocular 3D Face Reconstruction from Dense UV-space Priors
🤖 AI Summary
This work addresses the high computational cost and lack of real-time performance in traditional optimization-based monocular 3D face reconstruction. The authors formulate 3D Morphable Model (3DMM) fitting as a nonlinear least-squares problem and develop a customized Gauss-Newton solver that converges within only a few iterations. To guide this efficient two-stage optimization, they introduce a deep network that predicts UV correspondence and relative depth maps from a single RGB image, providing dense priors. This approach achieves, for the first time, real-time optimization-based reconstruction leveraging dense UV priors, supporting three operational modes: single-image, offline sequence, and online tracking. Evaluated on the NeRSemble SVFR benchmark, the method attains state-of-the-art accuracy, achieves over 80 FPS in online tracking, and accelerates offline processing by more than 20× compared to prior optimization-based methods.
📝 Abstract
Recent monocular 3D face reconstruction methods achieve high fidelity by fitting a 3D Morphable Model (3DMM) to dense priors predicted by networks, but the optimization stage is computationally expensive, often taking tens of seconds per image. We present RealDenseFace, a real-time optimization-based 3D face reconstruction method with dense UV-space network predictions. Our key idea is to formulate 3DMM fitting as a nonlinear least-squares problem and solve it with a tailored Gauss-Newton solver that converges in only a few iterations. The reconstruction is conducted in two stages. In the first stage, the network predicts two dense UV-space maps from a single RGB image: a correspondence map for UV-to-image alignment, and a relative-depth map for geometric constraints along the viewing direction. In the second stage, the solver fits per-vertex targets sampled from these maps at the vertex UV coordinates. The solver supports all three reconstruction settings: single-image fitting, offline sequence reconstruction, and online tracking. Our method achieves state-of-the-art accuracy on the NeRSemble SVFR benchmark. The online tracker runs at 80+ FPS, and the offline sequence reconstruction is over 20 times faster than previous optimization-based baselines.
Innovation
Methods, ideas, or system contributions that make the work stand out.
real-time 3D face reconstruction
dense UV-space priors
nonlinear least-squares optimization
Gauss-Newton solver
3D Morphable Model (3DMM)
🔎 Similar Papers
No similar papers found.
