OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention

2026-05-07

AI Summary

This work addresses the fragmented and spatially inconsistent semantic predictions in open-vocabulary 3D scene understanding based on Gaussian representations. To this end, it proposes a Gaussian Feature Field that models semantics as a continuous function of Gaussian geometry and appearance. By introducing a structured semantic codebook together with a codebook-guided attention mechanism, the method achieves strong coupling between geometry and semantics while enforcing object-level consistency. Integrating 3D Gaussian splatting, continuous semantic modeling, and open-vocabulary language alignment, the approach significantly outperforms existing methods on both 2D and 3D open-vocabulary benchmarks. It not only enhances segmentation quality and 3D semantic coherence but also yields an interpretable semantic codebook representation.

Abstract

Understanding open-vocabulary 3D scenes with Gaussian-based representations remains challenging due to fragmented and spatially inconsistent semantic predictions across multi-view observations. In this paper, we present OpenGaFF, a novel framework for open-vocabulary 3D scene understanding built upon 3D Gaussian Splatting. At the core of our method is a Gaussian Feature Field that models semantics as a continuous function of Gaussian geometry and appearance. By explicitly conditioning semantic predictions on geometric structure, this formulation strengthens the coupling between geometry and semantics, leading to improved spatial coherence across similar structures in 3D space. To further enforce object-level semantic consistency, we introduce a structured codebook that serves as a set of shared semantic primitives. Furthermore, a codebook-guided attention mechanism is proposed to retrieve language features via similarity matching between query embeddings and learned codebook entries, enabling robust open-vocabulary reasoning while reducing intra-object feature variance. Extensive experiments on standard 2D and 3D open-vocabulary benchmarks demonstrate that our method consistently outperforms prior approaches, achieving improved segmentation quality, stronger 3D semantic consistency and a semantically interpretable codebook that provides insight into the learned representation.

Problem

• open-vocabulary
• 3D scene understanding
• semantic consistency
• Gaussian representation
• multi-view observations

Innovation

• Gaussian Feature Field
• codebook attention
• open-vocabulary 3D understanding
• semantic consistency
• 3D Gaussian Splatting