SparseTalk - Sparsifying 3D Gaussian Language Fields for Efficient 3D Visual Question Answering

2026-09-14

AI Summary

研究通过稀疏化3D高斯语言场解决3D视觉问答中的存储和计算成本问题，提出基于对象的稀疏化方法，实验证明仅需少量语义嵌入即可保持性能。

Abstract

3D Gaussian language fields provide an explicit, spatially grounded representation for 3D visual question answering (VQA), but their dense semantic features can require tens of thousands of embeddings per scene, resulting in substantial storage, memory, and inference costs. We investigate how much of this representation is actually necessary for downstream reasoning. Starting from a full embedding representation, we systematically sparsify its semantic embeddings, including the previously underexplored regime below a single image-equivalent block down to 8 visual tokens. We compare random, geometric, semantic, and joint spatial-semantic selection strategies and introduce an object-based sparsification method that distributes the token budget across detected object instances while retaining background context. Experiments on ScanQA and MV-ScanQA reveal substantial redundancy in dense Gaussian language fields. Strong VQA performance is retained with only a few hundred semantic embeddings, corresponding to less than 1% of the original representation. Object-based selection performs well relative to others, with only modest observed changes down to 256 tokens. At this budget, SparseTalk retains 0.80% of SplatTalk's 32,076-token inference input and 0.332% of the mean 77,207-Gaussian dense field, increasing inference throughput while reducing decoded-feature memory 125-fold.

Problem

• 3D Gaussian language fields
• 3D visual question answering
• semantic embeddings
• sparsification
• inference costs

Innovation

• Sparsification
• 3D Gaussian Language Fields
• Visual Question Answering
• Object-based Selection
• Efficient Inference