JacobianAvatar: Temporally Consistent Semi-rigid Avatar Reconstruction from a Monocular Video

2026-06-30

AI Summary

Reconstructing photorealistic human avatars with complex dynamics—such as cloth motion—from monocular video is highly challenging, as it requires simultaneous modeling of global and local deformations. This work proposes a self-supervised method based on Neural Jacobian Fields (NJF) that reconstructs animatable human avatars by predicting pose-dependent Jacobian matrices and solving a Poisson equation. Key innovations include a constrained Poisson solver to suppress boundary artifacts, Jacobian regularization grounded in signed distance fields to recover occluded regions, and a deformation-guided residual optical flow loss to enforce temporal consistency. Evaluated on both benchmark and in-the-wild videos, the method produces geometrically coherent and temporally stable avatars, outperforming current state-of-the-art approaches.

Abstract

Generating realistic human avatars in complex motions--such as clothing dynamics--requires modeling of global and local deformations which remains challenging in monocular settings. We address this problem by leveraging neural Jacobian fields (NJFs) for representing semi-rigid deformations. We train self-supervised neural networks for predicting Jacobian matrices that give the pose-dependent deformations, by solving a Poisson equation. However, monocular input presents several difficulties such as self-occluded regions and invisible surfaces. To address these issues, we introduce three key components: a constrained Poisson solver, signed distance-based Jacobian regularization, and a deformation-guided residual flow loss, which together suppress boundary artifacts, recover frequently occluded regions such as armpits and thighs, and enforce temporal consistency during motion. Experiments on benchmark and in-the-wild videos demonstrate that our method generates temporally stable and geometrically coherent avatars, outperforming state-of-the-art approaches.

Problem

• monocular video
• avatar reconstruction
• temporal consistency
• semi-rigid deformation
• self-occlusion

Innovation

• Neural Jacobian Fields
• Monocular Avatar Reconstruction
• Temporal Consistency
• Semi-rigid Deformation
• Self-supervised Learning