$R^3$: 3D Reconstruction via Relative Regression

2026-05-26

AI Summary

This work addresses the limitations of existing feedforward geometric foundation models, which rely on a global coordinate system and struggle with unbounded pose drift in long-horizon or streaming 3D reconstruction scenarios, often requiring an arbitrary temporal origin. To overcome this, the authors propose a relative regression mechanism that dispenses with the global coordinate assumption. Their approach employs a lightweight MLP to predict confidence-weighted relative pose constraints and introduces a confidence-aware loss during training, along with a confidence-guided pose aggregation strategy at inference. This framework uniformly supports both offline and causally constrained streaming reconstruction, significantly enhancing stability and scalability in long-sequence 3D reconstruction under both settings.

Abstract

Recent feed-forward geometry foundation models have demonstrated impressive generalization by recovering depth and poses in a single forward pass. However, these models are typically constrained by a global coordinate frame assumption. This dependency becomes a significant bottleneck for long-context and streaming reconstruction, as it forces the network to maintain an arbitrary temporal origin and handle translation magnitudes that grow unbounded over time. Our solution, which we call $R^3$, employs relative regression. We employ a lightweight MLP to predict confidence-weighted relative constraints. These confidences serve as a unified anchor: weighting losses during training and guiding pose aggregation during inference. $R^3$ supports both full-context offline reconstruction and causal, bounded-memory streaming. Our evaluation in both offline and streaming settings validates the effectiveness of our relative mechanism. Project page: https://kevinxu02.github.io/r3-site

Problem

• 3D reconstruction
• global coordinate frame
• streaming reconstruction
• long-context
• pose estimation

Innovation

• relative regression
• 3D reconstruction
• streaming reconstruction
• pose aggregation
• confidence-weighted constraints