Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs

2026-06-30

AI Summary

This work addresses the limitation of existing open-vocabulary 3D scene graph methods, which rely on a sequential “reconstruct-then-enrich” pipeline that precludes real-time querying during exploration. To overcome this, the authors propose an asynchronous architecture that decouples lightweight online mapping from heavyweight semantic refinement, executing them in parallel. A probabilistic voxel backbone maintains object identity consistency, while a background visual-language model (VLM) agent incrementally enriches semantic annotations. The approach further incorporates semantic loop closure to eliminate redundant trajectories and introduces a multi-objective frame scheduler to reduce VLM computational overhead. This is the first method to enable queryable scene graphs during active exploration, achieving state-of-the-art semantic segmentation performance on ScanNet and Replica, and significantly outperforming prior art by 15.3–18.8 A@0.25 on the Sr3D+, Nr3D, and ScanRefer benchmarks.

Abstract

Open-vocabulary 3D scene graph methods typically operate in two stages: first reconstruct, then enrich with vision-language models, leaving the graph unqueryable during exploration. We argue that this sequential coupling is unnecessary and propose an asynchronous architecture in which lightweight online mapping runs concurrently with heavyweight semantic refinement. A probabilistic voxel-based backbone maintains stable object identities incrementally, while background VLM agents progressively enrich the graph. This framework resolves duplicate object tracks through semantic loop closure, attaches fine-grained visual attributes and derives spatial relations between objects. A multi-target frame scheduler amortizes VLM cost by selecting a small set of informative frames that jointly cover multiple targets. The resulting scene graph is queryable during exploration and grows in semantic richness over time. Our method matches or outperforms existing open-vocabulary 3D scene graph methods on semantic segmentation (ScanNet, Replica) and surpasses the prior state-of-the-art across three visual grounding benchmarks (Sr3D+, Nr3D, ScanRefer) by 15.3 to 18.8 A@0.25. Project page: https://denizbickici.github.io/thinkgraphs/

Problem

• open-vocabulary 3D scene graph
• incremental mapping
• real-time queryability
• semantic enrichment
• asynchronous processing

Innovation

• asynchronous architecture
• incremental 3D scene graph
• vision-language models
• semantic loop closure
• multi-target frame scheduling