ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

2026-07-06

AI Summary

In streaming 3D reconstruction, recursive updates are highly susceptible to noise or ambiguous observations, leading to degradation of historical information. This work proposes a plug-and-play reliability-calibrated learning rate mechanism that, for the first time, integrates token-level reliability estimation with dynamic learning rate adjustment. The method generates candidate learning rates by leveraging token alignment, state reconstruction residuals, and recent update pressure, then computes the final token-wise learning rate through reliability-weighted interpolation. This approach effectively suppresses aggressive updates in unreliable regions while preserving the model’s ability to adapt to informative frames. Evaluated on CUT3R, the proposed method reduces absolute trajectory error (ATE) by 3.7×, significantly improving pose accuracy, depth estimation, and reconstruction quality without incurring additional runtime or memory overhead.

Abstract

Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7$\times$ reduction in ATE, with comparable runtime and memory. Code is available at: https://github.com/Powertony102/ReCal3R.

Problem

• streaming 3D reconstruction
• recurrent scene state
• state corruption
• reliability calibration
• learning rate

Innovation

• reliability-calibrated learning rate
• streaming 3D reconstruction
• recurrent scene representation
• token-wise adaptation
• training-free calibration