Shape-from-Template with Generalised Camera

2025-08-19

AI Summary

This work addresses the non-rigid Shape-from-Template (SfT) problem in multi-camera systems. To overcome the limitations of single-view SfT—particularly in medical imaging and handheld applications—the paper proposes the first multi-view SfT framework based on a generalized camera model, supporting both perspective and orthographic projections. The method introduces inter-view geometric constraints to jointly optimize keypoint correspondences and 3D non-rigid deformations: it computes an initial correspondence solution via convex optimization and iteratively refines it using silhouette information. Crucially, it accommodates directional vectors derived from either known or unknown 3D points, enabling broad applicability. Extensive experiments on synthetic and real-world datasets demonstrate significant improvements in accuracy and robustness for multi-view non-rigid shape reconstruction compared to prior approaches.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

This article presents a new method for non-rigidly registering a 3D shape to 2D keypoints observed by a constellation of multiple cameras. Non-rigid registration of a 3D shape to observed 2D keypoints, i.e., Shape-from-Template (SfT), has been widely studied using single images, but SfT with information from multiple-cameras jointly opens new directions for extending the scope of known use-cases such as 3D shape registration in medical imaging and registration from hand-held cameras, to name a few. We represent such multi-camera setup with the generalised camera model; therefore any collection of perspective or orthographic cameras observing any deforming object can be registered. We propose multiple approaches for such SfT: the first approach where the corresponded keypoints lie on a direction vector from a known 3D point in space, the second approach where the corresponded keypoints lie on a direction vector from an unknown 3D point in space but with known orientation w.r.t some local reference frame, and a third approach where, apart from correspondences, the silhouette of the imaged object is also known. Together, these form the first set of solutions to the SfT problem with generalised cameras. The key idea behind SfT with generalised camera is the improved reconstruction accuracy from estimating deformed shape while utilising the additional information from the mutual constraints between multiple views of a deformed object. The correspondence-based approaches are solved with convex programming while the silhouette-based approach is an iterative refinement of the results from the convex solutions. We demonstrate the accuracy of our proposed methods on many synthetic and real data

Problem

• Non-rigid 3D shape registration to 2D keypoints
• Using multiple camera generalized camera model
• Improving reconstruction accuracy with multi-view constraints

Innovation

• Generalised camera model for multi-camera setup
• Convex programming for correspondence-based registration
• Iterative refinement using silhouette information