GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generator

2026-03-26

AI Summary

This work addresses floating artifacts, flickering, and blurriness in 3D Gaussian splatting reconstructions of wild scenes, which arise from camera pose errors, insufficient coverage, and noisy geometric initialization. To resolve these issues, the authors propose a geometry-guided video-to-video generation approach that refines rendered outputs with temporal consistency. Their method introduces, for the first time, a geometry-aware video generation framework that constructs a Gaussian primitive video buffer using depth, normals, opacity, and covariance. Combined with a synthetic data training strategy capable of simulating diverse degradation patterns, this approach significantly enhances generalization. The method achieves state-of-the-art performance on novel view synthesis benchmarks, with an efficient variant running at 21 FPS, enabling interactive applications.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

We present GaussFusion, a novel approach for improving 3D Gaussian splatting (3DGS) reconstructions in the wild through geometry-informed video generation. GaussFusion mitigates common 3DGS artifacts, including floaters, flickering, and blur caused by camera pose errors, incomplete coverage, and noisy geometry initialization. Unlike prior RGB-based approaches limited to a single reconstruction pipeline, our method introduces a geometry-informed video-to-video generator that refines 3DGS renderings across both optimization-based and feed-forward methods. Given an existing reconstruction, we render a Gaussian primitive video buffer encoding depth, normals, opacity, and covariance, which the generator refines to produce temporally coherent, artifact-free frames. We further introduce an artifact synthesis pipeline that simulates diverse degradation patterns, ensuring robustness and generalization. GaussFusion achieves state-of-the-art performance on novel-view synthesis benchmarks, and an efficient variant runs in real time at 21 FPS while maintaining similar performance, enabling interactive 3D applications.

Problem

• 3D reconstruction
• 3D Gaussian splatting
• artifacts
• camera pose errors
• temporal coherence

Innovation

• 3D Gaussian Splatting
• geometry-informed video generation
• temporal coherence
• artifact synthesis
• real-time rendering