Multimodal-Prior-Guided Importance Sampling for Hierarchical Gaussian Splatting in Sparse-View Novel View Synthesis

2026-03-03

AI Summary

This work addresses the challenge of recovering fine geometric details in novel view synthesis from sparse inputs, where existing methods often suffer from texture overfitting and inconsistencies in pose or appearance. To mitigate these issues, we propose a multimodal prior-guided importance sampling mechanism that integrates photometric residuals, semantic cues, and geometric priors to assess local recoverability. Guided by this assessment, our method selectively injects refined Gaussian points into a hierarchical 3D Gaussian splatting representation. By leveraging multimodal priors to inform the placement of new Gaussians—rather than relying solely on residual errors—we effectively avoid overfitting. Furthermore, we introduce a geometry-aware strategy for adding and retaining Gaussians, which preserves newly introduced points in under-constrained regions. Our approach achieves state-of-the-art performance across multiple sparse-view benchmarks, with up to a 0.3 dB PSNR improvement on the DTU dataset.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present multimodal-prior-guided importance sampling as the central mechanism for hierarchical 3D Gaussian Splatting (3DGS) in sparse-view novel view synthesis. Our sampler fuses complementary cues { -- } photometric rendering residuals, semantic priors, and geometric priors { -- } to produce a robust, local recoverability estimate that directly drives where to inject fine Gaussians. Built around this sampling core, our framework comprises (1) a coarse-to-fine Gaussian representation that encodes global shape with a stable coarse layer and selectively adds fine primitives where the multimodal metric indicates recoverable detail; and (2) a geometric-aware sampling and retention policy that concentrates refinement on geometrically critical and complex regions while protecting newly added primitives in underconstrained areas from premature pruning. By prioritizing regions supported by consistent multimodal evidence rather than raw residuals alone, our method alleviates overfitting texture-induced errors and suppresses noise from pose/appearance inconsistencies. Experiments on diverse sparse-view benchmarks demonstrate state-of-the-art reconstructions, with up to +0.3 dB PSNR on DTU.

Problem

• sparse-view novel view synthesis
• 3D Gaussian Splatting
• importance sampling
• multimodal priors
• overfitting

Innovation

• multimodal-prior-guided sampling
• hierarchical Gaussian Splatting
• sparse-view novel view synthesis
• geometric-aware refinement
• importance sampling