LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS

2025-07-08

AI Summary

LangSplat enables precise 3D language field modeling by embedding CLIP textual features into 3D Gaussian splatting space, but suffers from low inference speed (8.2 FPS), hindering real-time language interaction. To address this, we propose a sparse 3D language Gaussian lattice framework featuring a learnable 3D sparsity coefficient field and a lightweight decoder-free encoding mechanism that jointly integrates CLIP semantic embeddings, SAM segmentation priors, and CUDA-level parallel optimization. On an A100 GPU, our method achieves 476.2 FPS for high-dimensional feature rendering and 384.6 FPS for open-vocabulary 3D text queries—47× faster than LangSplat—while improving accuracy and enabling real-time, language-driven interaction in complex scenes. Our core contribution is the first real-time rendering and retrieval of high-dimensional semantic Gaussian lattices, effectively breaking the efficiency bottleneck in language-guided 3D understanding.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this paper, we introduce LangSplatV2, which achieves high-dimensional feature splatting at 476.2 FPS and 3D open-vocabulary text querying at 384.6 FPS for high-resolution images, providing a 42 $ imes$ speedup and a 47 $ imes$ boost over LangSplat respectively, along with improved query accuracy. LangSplat employs Gaussian Splatting to embed 2D CLIP language features into 3D, significantly enhancing speed and learning a precise 3D language field with SAM semantics. Such advancements in 3D language fields are crucial for applications that require language interaction within complex scenes. However, LangSplat does not yet achieve real-time inference performance (8.2 FPS), even with advanced A100 GPUs, severely limiting its broader application. In this paper, we first conduct a detailed time analysis of LangSplat, identifying the heavyweight decoder as the primary speed bottleneck. Our solution, LangSplatV2 assumes that each Gaussian acts as a sparse code within a global dictionary, leading to the learning of a 3D sparse coefficient field that entirely eliminates the need for a heavyweight decoder. By leveraging this sparsity, we further propose an efficient sparse coefficient splatting method with CUDA optimization, rendering high-dimensional feature maps at high quality while incurring only the time cost of splatting an ultra-low-dimensional feature. Our experimental results demonstrate that LangSplatV2 not only achieves better or competitive query accuracy but is also significantly faster. Codes and demos are available at our project page: https://langsplat-v2.github.io.

Problem

• Achieve real-time 3D language feature splatting at 450+ FPS
• Eliminate heavyweight decoder bottleneck in 3D language fields
• Enable efficient open-vocabulary text querying in complex 3D scenes

Innovation

• 3D sparse coefficient field eliminates heavyweight decoder
• Efficient sparse splatting with CUDA optimization
• High-dimensional feature rendering at ultra-low cost