Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors

2025-03-21

AI Summary

To address the low utilization efficiency and poor generalization of multi-source geometric priors (e.g., camera intrinsics, poses, sparse/dense depth) in unconstrained 3D reconstruction, this paper proposes the first end-to-end regression framework supporting modality-adaptive inference. Methodologically, it dynamically fuses heterogeneous priors within a unified model via multimodal conditional encoding and stochastic modality masking during training, enabling flexible runtime adaptation to arbitrary prior combinations. Built upon the lightweight DUSt3R architecture, it supports native-resolution image inference and novel tasks such as point cloud completion. Crucially, it adopts a unified regression-based 3D representation learning paradigm. The framework achieves state-of-the-art performance across five core tasks—3D reconstruction, depth completion, multi-view depth prediction, multi-view stereo (MVS), and pose estimation—demonstrating significantly improved prior utilization efficiency and cross-scene generalization capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present Pow3r, a novel large 3D vision regression model that is highly versatile in the input modalities it accepts. Unlike previous feed-forward models that lack any mechanism to exploit known camera or scene priors at test time, Pow3r incorporates any combination of auxiliary information such as intrinsics, relative pose, dense or sparse depth, alongside input images, within a single network. Building upon the recent DUSt3R paradigm, a transformer-based architecture that leverages powerful pre-training, our lightweight and versatile conditioning acts as additional guidance for the network to predict more accurate estimates when auxiliary information is available. During training we feed the model with random subsets of modalities at each iteration, which enables the model to operate under different levels of known priors at test time. This in turn opens up new capabilities, such as performing inference in native image resolution, or point-cloud completion. Our experiments on 3D reconstruction, depth completion, multi-view depth prediction, multi-view stereo, and multi-view pose estimation tasks yield state-of-the-art results and confirm the effectiveness of Pow3r at exploiting all available information. The project webpage is https://europe.naverlabs.com/pow3r.

Problem

• Enhancing 3D reconstruction with camera and scene priors
• Integrating multiple input modalities in a single network
• Improving accuracy in 3D vision tasks with auxiliary data

Innovation

• Incorporates multiple input modalities in one network
• Uses transformer-based architecture with pre-training
• Random subset training for versatile inference