Beyond Similarity Matching: Structured Reasoning for Open-Vocabulary Referring Segmentation in 3DGS

2026-08-17

AI Summary

This study addresses semantic confusion and granularity mismatch caused by global similarity in open-vocabulary referring segmentation for 3D Gaussian Splatting (3DGS). We propose QAGaussian, a novel framework featuring query-conditioned multi-scale slot learning and relation-aware graph reasoning. By integrating granularity-adaptive routing with relation-constrained refinement, the method enables precise language-guided selection and structured reasoning of Gaussian primitives. Experimental results demonstrate that QAGaussian achieves an average mIoU of 47.2 and an F1 score of 63.2, significantly outperforming state-of-the-art baselines. These findings confirm the model’s effectiveness in mitigating target confusion and substantially improving part-level segmentation accuracy within 3DGS-based open-vocabulary referring segmentation tasks.

Abstract

Open-vocabulary referring segmentation in 3D Gaussian Splatting (3DGS) requires a neural model to select Gaussian primitives according to free-form language expressions. Existing 3DGS-based methods usually rely on global text-region similarity, which is weak for queries involving attributes, reference objects, spatial relations, and fine-grained parts. This often causes target-reference confusion, granularity mismatch, part-whole leakage, and relation violations. We propose QAGaussian, a query-adaptive neural reasoning framework for language-guided Gaussian primitive selection. QAGaussian first learns query-conditioned multi-scale Gaussian slots as differentiable candidates whose receptive fields are shaped by the input expression. It then builds a relation-aware slot graph with language-conditioned edge weighting to propagate target-reference, attribute, part-whole, and contextual evidence. A granularity-adaptive router softly combines region-level, object-level, part-level, attribute-aware, and relation-aware mask branches, followed by relation-constrained refinement for spatial, part-whole, attribute, and geometric consistency. QAGaussian is pretrained only on Mosaic3D-5.6M for Gaussian-text alignment and evaluated on independent benchmarks without target-dataset fine-tuning. It achieves 47.2 Avg. mIoU and 63.2 Avg. F1, outperforming the strongest 3DGS referring baseline by 2.7 mIoU points and 2.9 F1 points. It also improves Part-mIoU from 38.6 to 43.4, Rel-mIoU from 44.4 to 50.8, and reduces target-reference confusion from 10.8 to 7.4. These results demonstrate that query-conditioned slot learning, relation-aware graph reasoning, and adaptive routing provide an effective neural modeling strategy for open-vocabulary referring segmentation in 3DGS. The code is available at https://github.com/zqeslwyz/QAGaussian.

Problem

• Open-Vocabulary Referring Segmentation
• 3D Gaussian Splatting
• Structured Reasoning
• Target-Reference Confusion
• Granularity Mismatch

Innovation

• Query-adaptive Reasoning
• 3D Gaussian Splatting
• Relation-aware Graph
• Open-vocabulary Referring Segmentation
• Granularity-adaptive Router