Relightable Gaussian Splatting for Virtual Production Using Image-Based Illumination

2026-05-09

AI Summary

This work addresses the limitations of virtual production, where LED wall backgrounds are tightly coupled with lighting, restricting relighting flexibility in post-production, and where traditional inverse rendering based on environment maps lacks accuracy under near-field, high-resolution illumination. To overcome these challenges, the authors propose a relightable Gaussian splatting framework tailored for virtual production that dispenses with environment maps entirely. Instead, background textures are sampled directly in image space using Gaussian primitives modulated by UV coordinates, intensity, and resolution, implicitly modeling reflections and refractions. Leveraging known background information, relighting is guided and reduced to an image editing task. By decomposing appearance and lighting from multi-illumination real-world data, the method achieves high-quality reconstruction and controllable relighting at ~35 FPS with under two hours of training and less than 5 GB of GPU memory, while supporting outputs such as depth, light intensity, color, and unlit renderings.

Abstract

Virtual production (VP) use LED walls to provide both background imagery and image-based lighting. While this enables on-set compositing, it couples lighting to background and scene appearance, limiting flexibility for downstream editing. In addition, inverse rendering conventionally relies on physically-based rendering to estimates 3D geometry and lighting, using environment maps. However, these maps are typically low-resolution and assume far-field lighting. In VP, with near-field and high-resolution image-based lighting, this can lead to inaccuracies and introduce complexities when editing. Addressing this, we propose a VP-specific framework for 3D reconstruction and relighting using Gaussian Splatting. This uses the known background imagery to condition the relighting process. This avoids relying on environment maps and reduces compositing to a background-image editing task. To realize our framework, we introduce a process (and associated dataset) that captures real VP scenes under varying background content and illumination conditions. This data is used to decompose a 3D scene into fixed appearance and variable lighting components. The variable lighting process simulates light transport by parameterizing each primitive with a UV coordinate, intensity value and resolution modifier. Using mipmaps, these directly sample the background texture in image space - implicitly capturing reflections and refractions without physically-based rendering. Combined with the fixed appearance component, this allows us to render relit scenes using a Gaussian Splatting rasterizer. Compared to baselines, our approach achieves higher-quality 3D reconstruction and controllable relighting. The method is efficient (<3 GB RAM, <5 GB VRAM, <2 hours training, ~35 FPS) and supports rendering useful arbitrary output variables including depth, lighting intensity, lighting color, and unlit renders.

Problem

• virtual production
• image-based illumination
• relighting
• environment maps
• near-field lighting

Innovation

• Relightable Gaussian Splatting
• Image-Based Illumination
• Virtual Production
• Light Transport Simulation
• Mipmap Sampling