ShapeR: Robust Conditional 3D Shape Generation from Casual Captures

2026-01-16

AI Summary

This work proposes a robust 3D shape generation framework designed for in-the-wild conditions, where existing methods often fail due to their reliance on clean, precisely segmented inputs. The approach integrates sparse SLAM point clouds, multi-view images, and machine-generated textual descriptions, leveraging a specially trained rectified flow Transformer to achieve high-fidelity metric 3D reconstruction. Key innovations include dynamic composition augmentation, cross-dataset curriculum training, and background interference suppression. The authors also introduce the first benchmark comprising 178 real-world objects captured under uncontrolled conditions. Experimental results demonstrate that the proposed method significantly outperforms current state-of-the-art techniques in real-world scenarios, achieving a 2.7× improvement in Chamfer distance.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Recent advances in 3D shape generation have achieved impressive results, but most existing methods rely on clean, unoccluded, and well-segmented inputs. Such conditions are rarely met in real-world scenarios. We present ShapeR, a novel approach for conditional 3D object shape generation from casually captured sequences. Given an image sequence, we leverage off-the-shelf visual-inertial SLAM, 3D detection algorithms, and vision-language models to extract, for each object, a set of sparse SLAM points, posed multi-view images, and machine-generated captions. A rectified flow transformer trained to effectively condition on these modalities then generates high-fidelity metric 3D shapes. To ensure robustness to the challenges of casually captured data, we employ a range of techniques including on-the-fly compositional augmentations, a curriculum training scheme spanning object- and scene-level datasets, and strategies to handle background clutter. Additionally, we introduce a new evaluation benchmark comprising 178 in-the-wild objects across 7 real-world scenes with geometry annotations. Experiments show that ShapeR significantly outperforms existing approaches in this challenging setting, achieving an improvement of 2.7x in Chamfer distance compared to state of the art.

Problem

• 3D shape generation
• casual captures
• robustness
• real-world scenarios
• conditional generation

Innovation

• conditional 3D generation
• casual captures
• rectified flow transformer
• multimodal conditioning
• robust 3D reconstruction