DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term Scene Understanding and Grounding

2026-05-28

AI Summary

This work addresses the limitations of existing methods in long-term scene understanding, which struggle to robustly associate instances, handle object-level topological changes, and either lack explicit spatial reasoning or rely on offline ground-truth geometry. To overcome these challenges, we propose the 3D Gaussian Mind architecture, which uniquely integrates dynamic 3D Gaussians with a hierarchical scene graph. Our approach combines probabilistic voxel grids, geometric-semantic consistency constraints, local mask refinement, and Gaussian relocation to enable cross-modal instance fusion and incremental semantic mapping. The resulting system supports zero-shot 3D visual localization, open-vocabulary semantic segmentation, and real-time dynamic updates. Evaluated on self-reconstructed maps, it achieves state-of-the-art localization performance and demonstrates embodied, goal-directed multimodal reasoning capabilities on real robotic platforms.

Abstract

Integrating open-vocabulary semantic information into dynamic 3D scene representations is essential for long-term embodied scene understanding. However, existing methods often suffer from fragile instance association due to incomplete cross-view cues, while their limited ability to handle object-level topological changes restricts long-term robotic task execution. Moreover, current 3D scene understanding methods either rely on simple feature matching without explicit spatial reasoning or assume offline ground-truth 3D geometry. To address these challenges, we present DGSG-Mind, a hybrid instance-aware 3D Gaussian dynamic scene graph system with an embodied reasoning agent. Our system couples a probabilistic voxel grid with explicit 3D Gaussians to enable robust cross-modal instance fusion and incremental semantic mapping. It handles dynamic changes through Gaussian-based visual relocalization and localized masked refinement guided by geometric-semantic consistency. Built on the instance Gaussian map, DGSG-Mind further constructs a hierarchical scene graph and develops the 3D Gaussian Mind, which integrates structural relations, spatial-semantic information, and visually annotated RoI Gaussian renderings for multimodal reasoning. Extensive experiments show that DGSG-Mind achieves the best zero-shot 3DVG performance among methods operating on self-reconstructed maps, while also delivering strong performance in 3D open-vocabulary semantic segmentation and scene reconstruction. We further deploy DGSG-Mind on real-world robots to demonstrate its target-oriented reasoning and dynamic update capabilities. The project page of DGSG-Mind is available at https://icr-lab.github.io/DGSG-Mind

Problem

• dynamic 3D scene understanding
• open-vocabulary semantics
• instance association
• topological changes
• spatial reasoning

Innovation

• Dynamic 3D Gaussian
• Scene Graph
• Embodied Reasoning
• Open-Vocabulary Semantics
• Incremental Semantic Mapping