GaussFusion: Towards Multimodal 3D Gaussian Pretraining

2026-07-07

AI Summary

This work addresses the limited transferability of existing pretraining methods for 3D Gaussian representations, which predominantly rely on local structural reconstruction and lack semantic supervision. To overcome this limitation, we propose GaussFusion, the first 3D Gaussian pretraining framework that incorporates image–text multimodal supervision. By aligning cross-modal representations, GaussFusion enables semantic modeling of masked Gaussians and introduces a multi-scale continuous hole masking strategy based on Gaussian saliency to enhance joint visual–linguistic understanding. Experimental results demonstrate that our method outperforms Gaussian-MAE by 0.61% on ModelNet40 and by 3.85% on ScanObjectNN (PB-T50-RS), significantly improving the semantic awareness and transferability of 3D Gaussian representations.

Abstract

3D Gaussian Splatting provides an explicit representation that jointly models geometry and appearance, serving as a scalable foundation for 3D representation learning. Existing pre-training methods for Gaussian representations, such as masked Gaussian reconstruction, primarily capture local structures but offer limited semantic supervision. In this paper, we propose GaussFusion, a multimodal pre-training framework for 3D Gaussian representations. GaussFusion integrates image and text supervision into masked Gaussian modeling through cross-modal semantic alignment, enabling the Gaussian encoder to learn both visual and language-level semantic information during pre-training. To better adapt masked modeling to the non-uniform distribution of Gaussian primitives, we further propose Gaussian Salience-guided Multi-scale Hole Masking (GSHM). GSHM constructs spatially continuous masked regions based on Gaussian salience. By applying hole masks at multiple scales, GSHM encourages the encoder to capture both fine-grained local patterns and broader structural dependencies. Extensive experiments on downstream tasks demonstrate that GaussFusion improves the transferability of Gaussian representations. Notably, GaussFusion outperforms Gaussian-MAE on ModelNet40 and ScanObjectNN (PB-T50-RS) by 0.61\% and 3.85\%, respectively.

Problem

• 3D Gaussian Splatting
• multimodal pre-training
• semantic supervision
• masked modeling
• 3D representation learning

Innovation

• multimodal pretraining
• 3D Gaussian Splatting
• cross-modal alignment
• masked modeling
• salience-guided masking