Ray-Aware Pointer Memory with Adaptive Updates for Streaming 3D Reconstruction

2026-05-07

AI Summary

Dense online 3D reconstruction from continuous image streams often suffers from inaccurate geometric aggregation and unstable long-term memory due to viewpoint variations, leading to redundant observations and drift. This work proposes a ray-aware pointer memory mechanism that jointly models 3D positions, observation ray directions, and feature embeddings, replacing conventional fusion-and-compression strategies with an adaptive update scheme. By introducing a spatial-ray joint similarity metric, the method unifies redundancy pruning, integration of novel observations, and online loop closure within a single framework. The approach effectively constrains memory growth while significantly improving global reconstruction consistency and camera pose accuracy, achieving scalable, drift-resistant online 3D reconstruction without compromising efficient streaming inference.

Abstract

Dense 3D reconstruction from continuous image streams requires both accurate geometric aggregation and stable long-term memory management. Recent feed-forward reconstruction frameworks integrate observations through persistent memory representations, yet most rely primarily on appearance-based similarity when updating memory. Such appearance-driven integration often leads to redundant accumulation of observations and unstable geometry when viewpoint changes occur. In this work, we propose a ray-aware pointer memory for streaming 3D reconstruction that explicitly models both spatial location and viewing direction within a unified memory representation. Each memory pointer stores its 3D position, associated ray direction, and feature embedding, allowing the system to reason jointly about geometric proximity and viewpoint consistency. Based on this representation, we introduce an adaptive pointer update strategy that replaces traditional fusion-based memory compression with a retain-or-replace mechanism. Instead of averaging nearby observations, the system selectively retains informative pointers while discarding redundant ones, preserving distinctive geometric structures while maintaining bounded memory growth. Furthermore, the joint reasoning over spatial distance and ray-direction discrepancy enables the system to distinguish between local redundancy, novel observations, and potential loop revisits in a unified manner. When loop candidates are detected, pose refinement is triggered to enforce global geometric consistency across the reconstruction. Extensive experiments demonstrate that the proposed ray-aware memory design significantly improves long-term reconstruction stability and camera pose accuracy while maintaining efficient streaming inference. Our approach provides a principled framework for scalable and drift-resistant online 3D reconstruction from image streams.

Problem

• streaming 3D reconstruction
• memory management
• geometric stability
• viewpoint consistency
• ray-aware representation

Innovation

• ray-aware memory
• adaptive pointer update
• streaming 3D reconstruction
• viewpoint consistency
• loop closure