BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video

2025-02-12

AI Summary

Traditional methods are constrained by fixed illumination, while neural approaches struggle to balance efficiency, quality, and generalizability. This paper introduces the first 4D Gaussian-based paradigm for relightable volumetric video synthesis, jointly reconstructing dynamic geometry and physically based rendering (PBR) material properties—including roughness, ambient occlusion, and base color—from multi-view RGB video sequences. We innovatively inject PBR attributes into 4D Gaussian voxels in a staged manner, design a multi-view diffusion-guided roughness generation module, and propose a Gaussian-accelerated 2D-to-3D attribute mapping framework. Our rendering pipeline integrates geometry-aware rasterization, customized Gaussian ray tracing, and hybrid deferred shading/path tracing. The method enables both real-time and offline relighting, preserving high-fidelity 4D geometric dynamics while significantly enhancing material realism and cross-view lighting consistency—fully compatible with standard computer graphics pipelines.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization.

Problem

• Relightable volumetric video generation
• Bridging Gaussian modeling and PBR
• High-quality 3D scene rendering

Innovation

• Bridges 4D Gaussian and PBR
• Uses Gaussian-based performance tracking
• Incorporates multi-view-conditioned diffusion model