Image-Guided Shape-from-Template Using Mesh Inextensibility Constraints

2025-07-30

AI Summary

Traditional Shape-from-Template (SfT) methods rely on image-to-template point correspondences, rendering them ineffective under severe occlusion. While recent unsupervised approaches eliminate the need for correspondence annotations, they suffer from slow inference and insufficient geometric detail recovery. This paper proposes a fully unsupervised SfT framework that reconstructs the template mesh from a single input image using only color, gradient, and contour observations. Our method performs end-to-end optimization by integrating differentiable rendering with a physics-based inextensibility constraint on the mesh surface as a geometric regularizer. It requires no point correspondences, external supervision, or pretraining. Quantitatively, our approach achieves a 400× speedup over the current state-of-the-art unsupervised method. Qualitatively and quantitatively, it demonstrates superior performance in reconstructing fine-grained geometry under strong occlusion and large non-rigid deformations.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Shape-from-Template (SfT) refers to the class of methods that reconstruct the 3D shape of a deforming object from images/videos using a 3D template. Traditional SfT methods require point correspondences between images and the texture of the 3D template in order to reconstruct 3D shapes from images/videos in real time. Their performance severely degrades when encountered with severe occlusions in the images because of the unavailability of correspondences. In contrast, modern SfT methods use a correspondence-free approach by incorporating deep neural networks to reconstruct 3D objects, thus requiring huge amounts of data for supervision. Recent advances use a fully unsupervised or self-supervised approach by combining differentiable physics and graphics to deform 3D template to match input images. In this paper, we propose an unsupervised SfT which uses only image observations: color features, gradients and silhouettes along with a mesh inextensibility constraint to reconstruct at a $400 imes$ faster pace than (best-performing) unsupervised SfT. Moreover, when it comes to generating finer details and severe occlusions, our method outperforms the existing methodologies by a large margin. Code is available at https://github.com/dvttran/nsft.

Problem

• Reconstruct 3D shapes from images without point correspondences
• Handle severe occlusions in image-guided shape reconstruction
• Achieve faster unsupervised 3D reconstruction with mesh constraints

Innovation

• Uses image features without correspondences
• Incorporates mesh inextensibility constraints
• Achieves 400x faster reconstruction speed