GaussianGrow: Geometry-aware Gaussian Growing from 3D Point Clouds with Text Guidance

2026-04-07

AI Summary

This work addresses the challenge of generating geometrically accurate 3D Gaussian representations from scratch in the absence of reliable geometric priors. It introduces, for the first time, a text-guided, progressive growth process that starts from sparse point clouds. The method leverages a multi-view diffusion model to provide appearance supervision and iteratively completes unobserved regions through 2D image inpainting, camera pose optimization, and dynamic viewpoint selection. A geometry-aware strategy is further integrated to enhance structural completeness. Experiments demonstrate that the proposed approach consistently produces high-quality, geometrically coherent, and semantically controllable 3D Gaussian scenes from both synthetic and real-world scanned point clouds, significantly outperforming existing text-to-3D generation methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

3D Gaussian Splatting has demonstrated superior performance in rendering efficiency and quality, yet the generation of 3D Gaussians still remains a challenge without proper geometric priors. Existing methods have explored predicting point maps as geometric references for inferring Gaussian primitives, while the unreliable estimated geometries may lead to poor generations. In this work, we introduce GaussianGrow, a novel approach that generates 3D Gaussians by learning to grow them from easily accessible 3D point clouds, naturally enforcing geometric accuracy in Gaussian generation. Specifically, we design a text-guided Gaussian growing scheme that leverages a multi-view diffusion model to synthesize consistent appearances from input point clouds for supervision. To mitigate artifacts caused by fusing neighboring views, we constrain novel views generated at non-preset camera poses identified in overlapping regions across different views. For completing the hard-to-observe regions, we propose to iteratively detect the camera pose by observing the largest un-grown regions in point clouds and inpainting them by inpainting the rendered view with a pretrained 2D diffusion model. The process continues until complete Gaussians are generated. We extensively evaluate GaussianGrow on text-guided Gaussian generation from synthetic and even real-scanned point clouds. Project Page: https://weiqi-zhang.github.io/GaussianGrow

Problem

• 3D Gaussian Splatting
• geometry-aware generation
• text-guided synthesis
• 3D point clouds
• geometric priors

Innovation

• 3D Gaussian Splatting
• point cloud
• text-to-3D
• diffusion model
• geometry-aware generation