RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

2025-06-05

AI Summary

Existing 3D shape completion methods suffer from geometric inconsistency, blurry boundaries, and high computational overhead. This paper proposes a zero-shot 3D object completion paradigm that reformulates completion as novel-view depth map synthesis, taking a single RGB-D image and arbitrary query rays as input and predicting depth, binary masks, and pixel-wise confidence scores in an end-to-end manner. Our key innovations include the first adoption of ray-level modeling with a feed-forward Transformer architecture, coupled with multi-view deep fusion and confidence-weighted reconstruction—enabling cross-category generalization without category-specific priors or fine-tuning. Evaluated on both synthetic and real-world datasets, our method achieves state-of-the-art performance, reducing 3D Chamfer distance by up to 44% over the best baseline. It simultaneously ensures geometric consistency, sharp boundary reconstruction, and efficient inference.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

3D shape completion has broad applications in robotics, digital twin reconstruction, and extended reality (XR). Although recent advances in 3D object and scene completion have achieved impressive results, existing methods lack 3D consistency, are computationally expensive, and struggle to capture sharp object boundaries. Our work (RaySt3R) addresses these limitations by recasting 3D shape completion as a novel view synthesis problem. Specifically, given a single RGB-D image and a novel viewpoint (encoded as a collection of query rays), we train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for those query rays. RaySt3R fuses these predictions across multiple query views to reconstruct complete 3D shapes. We evaluate RaySt3R on synthetic and real-world datasets, and observe it achieves state-of-the-art performance, outperforming the baselines on all datasets by up to 44% in 3D chamfer distance. Project page: https://rayst3r.github.io

Problem

• Predicting depth maps for 3D object completion
• Ensuring 3D consistency and sharp boundaries
• Achieving efficient novel view synthesis

Innovation

• Recasts 3D completion as view synthesis
• Uses transformer for depth and mask prediction
• Fuses multi-view predictions for 3D shapes