UniMGS: Unifying Mesh and 3D Gaussian Splatting with Single-Pass Rasterization and Proxy-Based Deformation
🤖 AI Summary
Existing methods struggle to unify the rendering of deformable meshes and 3D Gaussian Splatting (3DGS) due to incompatible representations and rendering mechanisms, leading to inaccurate occlusion and transparency handling, as well as deformation artifacts in 3DGS caused by topological flaws in proxy meshes. This work proposes UniMGS, the first framework enabling joint rendering of meshes and 3DGS within a single rasterization pass. By integrating anti-aliased alpha blending to fuse triangle and Gaussian fragments, and introducing a Gaussian-centric proxy mesh binding strategy, UniMGS achieves high-quality spatially consistent deformation. The method substantially reduces deformation artifacts, accurately resolves occlusion and transparency, and seamlessly integrates 3DGS into conventional mesh-based graphics pipelines, making it suitable for interactive applications such as embodied AI, virtual reality, and gaming.
Application Category
📝 Abstract
Joint rendering and deformation of mesh and 3D Gaussian Splatting (3DGS) have significant value as both representa tions offer complementary advantages for graphics applica tions. However, due to differences in representation and ren dering pipelines, existing studies render meshes and 3DGS separately, making it difficult to accurately handle occlusions and transparency. Moreover, the deformed 3DGS still suffers from visual artifacts due to the sensitivity to the topology quality of the proxy mesh. These issues pose serious obsta cles to the joint use of 3DGS and meshes, making it diffi cult to adapt 3DGS to conventional mesh-oriented graphics pipelines. We propose UniMGS, the first unified framework for rasterizing mesh and 3DGS in a single-pass anti-aliased manner, with a novel binding strategy for 3DGS deformation based on proxy mesh. Our key insight is to blend the col ors of both triangle and Gaussian fragments by anti-aliased {\alpha}-blending in a single pass, achieving visually coherent re sults with precise handling of occlusion and transparency. To improve the visual appearance of the deformed 3DGS, our Gaussian-centric binding strategy employs a proxy mesh and spatially associates Gaussians with the mesh faces, signifi cantly reducing rendering artifacts. With these two compo nents, UniMGS enables the visualization and manipulation of 3D objects represented by mesh or 3DGS within a unified framework, opening up new possibilities in embodied AI, vir tual reality, and gaming. We will release our source code to facilitate future research.
Problem
Research questions and friction points this paper is trying to address.
mesh
3D Gaussian Splatting
joint rendering
deformation
occlusion handling
Innovation
Methods, ideas, or system contributions that make the work stand out.
3D Gaussian Splatting
mesh deformation
single-pass rasterization
proxy-based binding
anti-aliased alpha blending
🔎 Similar Papers
No similar papers found.
