SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesis

2026-05-08

AI Summary

Existing general-purpose novel view synthesis methods employ fixed allocation strategies for Gaussian primitives, which struggle to adapt to spatial complexity variations across scenes, resulting in redundant resources in smooth regions and insufficient representation in detailed areas. This work proposes SplatWeaver, a framework that introduces dynamic primitive allocation within a feed-forward 3D Gaussian splatting architecture for the first time. By integrating a mixture-of-Gaussians expert model with a pixel-level routing mechanism—guided by high-frequency structural priors and enhanced through routing regularization—the method adaptively allocates Gaussian primitives according to local geometric complexity. Experiments demonstrate that SplatWeaver significantly outperforms state-of-the-art approaches using fewer Gaussians, consistently achieving higher rendering fidelity and improved detail reproduction across diverse scenes.

Abstract

Generalizable novel view synthesis aims to render unseen views from uncalibrated input images without requiring per-scene optimization. Recent feed-forward approaches based on 3D Gaussian Splatting have achieved promising efficiency and rendering quality. However, most of them assign a fixed number of Gaussians to each pixel or voxel, ignoring the spatially varying complexity of real-world scenes. Such uniform allocation often wastes Gaussian primitives in smooth regions while providing insufficient capacity for fine structures, complex geometry, and high-frequency details. This motivates us to predict region-dependent primitive cardinalities rather than impose a fixed primitive budget everywhere, enabling a more expressive yet compact 3D scene representation. Therefore, we propose SplatWeaver, a generalizable novel view synthesis framework that is able to dynamically allocate Gaussian primitives over different regions in a feed-forward manner. Specifically, SplatWeaver introduces cardinality Gaussian experts and a pixel-level routing scheme, wherein each expert specializes in producing a specific number of primitives from 0 to M, and the routing scheme coordinates these experts to adaptively determine how many Gaussian primitives should be allocated to each spatial location. Moreover, SplatWeaver incorporates a high-frequency prior with attendant guidance module and routing regularization to stabilize expert selection and promote complexity-aware allocation. By leveraging high-frequency structural cues, the routing process is encouraged to assign more Gaussian primitives to fine structures, complex geometry, and textured regions, while suppressing redundant primitives in smooth areas. Extensive experiments across diverse scenarios show that SplatWeaver consistently outperforms state-of-the-art methods, delivering more faithful novel-view renderings with fewer Gaussian primitives.

Problem

• generalizable novel view synthesis
• 3D Gaussian Splatting
• primitive allocation
• spatial complexity
• high-frequency details

Innovation

• dynamic Gaussian allocation
• generalizable novel view synthesis
• cardinality experts
• pixel-level routing
• high-frequency prior