PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models

2026-06-15

AI Summary

This work addresses the challenge of accurately registering RGB-only egocentric video sequences of the same indoor environment captured at different times, under conditions of motion blur, rapid camera movement, and partial scene overlap. The authors propose a zero-shot method that requires no training, depth sensors, or manual annotations, leveraging pretrained vision-language models (VLMs) and off-the-shelf foundation models to convert RGB sequences into object-level 3D scene graphs. Cross-sequence object instances are matched via semantic prompting, augmented with object height priors and intra- and inter-query verification, followed by geometric consistency optimization to refine the rigid transformation. This study presents the first application of VLMs to egocentric scene registration, outperforming both traditional geometric and learning-based approaches on the Aria Digital Twin and Aria Everyday Activities benchmarks, while producing scene graphs directly transferable to downstream tasks.

Abstract

Registering two captures of the same indoor space taken at different times underpins persistent spatial memory for robots and AR systems, yet the realistic version of this task is egocentric and its most scalable form is RGB-only. Head-mounted cameras yield blurry, fast-moving, partially overlapping views from which dense geometry is hard to recover. Classical registration leans on exactly the clean point clouds this setting lacks, while learned scene-graph methods require a pre-built or annotated graph and a trained matcher that we find brittle under egocentric data. We take a different route, using a pretrained vision-language model as the source of both scene understanding and cross-scan matching. Our method, PROSE (Prompted Scene rEgistration), lifts each RGB sequence into an object-level 3D scene graph using off-the-shelf foundation models for geometry, segmentation, and language, then prompts the same VLM to match object instances across the two RGB sequences. To make this matching tractable and reliable, we leverage object heights as a prior and verify each proposed match with a paired same/different query, then solve for the rigid transform by hypothesizing a candidate per matched object and selecting the one with the strongest geometric consensus. PROSE adds no learned parameters and requires no depth sensor, training, or annotated graph. On the egocentric Aria Digital Twin and Aria Everyday Activities benchmarks, it outperforms both geometric and learned scene-graph baselines in registration accuracy, on ground-truth and RGB-reconstructed point clouds alike, and the scene graph it produces transfers directly to downstream tasks.

Problem

• egocentric scene registration
• RGB-only
• spatial memory
• vision-language models
• indoor scene understanding

Innovation

• vision-language models
• egocentric scene registration
• training-free
• 3D scene graph
• RGB-only