OccGaussian: 3D Gaussian Splatting for Occluded Human Rendering

2024-04-12

arXiv.org

AI Summary

To address the challenges of occlusion-induced limb missing and inefficient real-time reconstruction for dynamic humans in monocular videos, this paper proposes the first occlusion-aware 3D Gaussian rasterization framework tailored for articulated human bodies. Methodologically, it initializes 3D Gaussians in a canonical space, introduces a pixel-aligned occlusion feature aggregation module, designs a Gaussian-feature MLP to jointly model visible and occluded regions, and formulates an occlusion-aware loss to enhance implicit completion capability. Experiments demonstrate that our method trains in only 6 minutes—250× faster than state-of-the-art (SOTA) approaches—and achieves 160 FPS inference speed—800× acceleration over SOTA. It matches or surpasses existing methods in PSNR and SSIM on both synthetic and real-world occlusion scenarios, significantly improving robustness and efficiency for real-time 3D human rendering from monocular video.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Hardware-aware ML

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Rendering dynamic 3D human from monocular videos is crucial for various applications such as virtual reality and digital entertainment. Most methods assume the people is in an unobstructed scene, while various objects may cause the occlusion of body parts in real-life scenarios. Previous method utilizing NeRF for surface rendering to recover the occluded areas, but it requiring more than one day to train and several seconds to render, failing to meet the requirements of real-time interactive applications. To address these issues, we propose OccGaussian based on 3D Gaussian Splatting, which can be trained within 6 minutes and produces high-quality human renderings up to 160 FPS with occluded input. OccGaussian initializes 3D Gaussian distributions in the canonical space, and we perform occlusion feature query at occluded regions, the aggregated pixel-align feature is extracted to compensate for the missing information. Then we use Gaussian Feature MLP to further process the feature along with the occlusion-aware loss functions to better perceive the occluded area. Extensive experiments both in simulated and real-world occlusions, demonstrate that our method achieves comparable or even superior performance compared to the state-of-the-art method. And we improving training and inference speeds by 250x and 800x, respectively. Our code will be available for research purposes.

Problem

• Real-time occluded human rendering
• Fast 3D Gaussian Splatting training
• High FPS with occlusion handling

Innovation

• 3D Gaussian Splatting
• Occlusion Feature Query
• Gaussian Feature MLP