QueryGaussian: Scalable and Training-Free Open-Vocabulary 3D Instance Retrieval

2026-06-17

AI Summary

This work addresses the challenge of excessive memory and computational costs in large-scale urban 3D scene understanding, where existing methods often suffer from GPU memory overflow due to linear scaling with scene complexity. The authors propose a training-free, open-vocabulary 3D instance retrieval framework that decouples semantic understanding from geometric representation through an instance-level querying mechanism. Leveraging pretrained 2D vision models to interpret textual prompts, the method lifts 2D segmentation masks into 3D space via a maximum-weight association strategy. It introduces, for the first time, a training-free paradigm for instance-level 3D retrieval, enhanced by multi-stage adaptive density clustering and a temporal fusion module to mitigate projection ambiguities. Achieving accuracy on par with state-of-the-art approaches, the method reduces GPU memory consumption by over 70% and accelerates inference by 180×, enabling efficient retrieval in city-scale scenes with tens of millions of Gaussians on consumer-grade GPUs.

Abstract

Efficiently retrieving specific 3D instances from large-scale scenes via natural language prompts remains a formidable challenge in multimedia analysis. Existing approaches predominantly follow a "scene-level embedding" paradigm, which requires distilling high-dimensional semantic features into every 3D primitive. This strategy suffers from a fundamental architectural bottleneck: memory and computational costs scale linearly with scene complexity, inevitably triggering out-of-memory (OOM) failures in city-scale environments. To address this barrier, we propose QueryGaussian, a training-free framework for expeditious and scalable open-vocabulary 3D instance retrieval. Unlike holistic semantic distillation, QueryGaussian employs an instance-level query mechanism that decouples semantic understanding from geometric representation. Specifically, we leverage pre-trained 2D vision models to interpret user prompts and lift segmentation masks into 3D via a concurrent maximum-weight association strategy, ensuring semantic-visual consistency. To mitigate projection ambiguity, we introduce a temporal fusion module with multi-stage adaptive density clustering. Experimental results demonstrate that QueryGaussian not only matches the accuracy of state-of-the-art methods but also delivers a decisive efficiency leap, reducing GPU memory usage by over 70% and accelerating inference by 180x. Crucially, QueryGaussian enables expeditious instance retrieval on city-scale scenes containing tens of millions of Gaussians using consumer-grade hardware.

Problem

• 3D instance retrieval
• open-vocabulary
• scalability
• memory efficiency
• large-scale scenes

Innovation

• training-free
• instance-level retrieval
• open-vocabulary 3D
• scalable 3D understanding
• 2D-to-3D lifting