Unsupervised 2D-3D lifting of non-rigid objects using local constraints

2025-04-27

AI Summary

This work addresses the ill-posed problem of unsupervised 3D shape reconstruction of non-rigid objects from single-frame 2D keypoints—particularly challenging under occlusion, viewpoint variation, and coupled deformations that induce geometric ambiguity. We propose a novel unsupervised learning framework whose core innovation is the first application of low-rank constraints to *local joint subsets* (rather than global shape), thereby embedding learnable geometric priors into deep models without relying on manually defined canonical spaces. The method integrates local low-rank regularization, reprojection consistency loss, and a self-supervised deformation decomposition module. Evaluated on the S-Up3D dataset, our approach reduces 3D reconstruction error by over 70% compared to prior unsupervised methods, achieving state-of-the-art performance.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

For non-rigid objects, predicting the 3D shape from 2D keypoint observations is ill-posed due to occlusions, and the need to disentangle changes in viewpoint and changes in shape. This challenge has often been addressed by embedding low-rank constraints into specialized models. These models can be hard to train, as they depend on finding a canonical way of aligning observations, before they can learn detailed geometry. These constraints have limited the reconstruction quality. We show that generic, high capacity models, trained with an unsupervised loss, allow for more accurate predicted shapes. In particular, applying low-rank constraints to localized subsets of the full shape allows the high capacity to be suitably constrained. We reduce the state-of-the-art reconstruction error on the S-Up3D dataset by over 70%.

Problem

• Predicting 3D shape from 2D keypoints for non-rigid objects
• Overcoming ill-posed issues due to occlusions and viewpoint changes
• Improving reconstruction quality with localized low-rank constraints

Innovation

• Unsupervised 2D-3D lifting with local constraints
• High capacity models improve shape accuracy
• Low-rank constraints on localized shape subsets