LumiMotion: Improving Gaussian Relighting with Scene Dynamics

2026-04-13

AI Summary

Existing inverse rendering methods based on Gaussian splatting struggle to disentangle material and lighting under complex real-world illumination, particularly due to their reliance on static scene assumptions, which limits their ability to handle the coupling between shadows and surface appearance. This work introduces dynamic information into the Gaussian splatting framework for the first time, leveraging motion in dynamic regions to provide multi-view lighting variations as a supervisory signal. We propose a dynamic 2D Gaussian splatting representation that models deformations in moving areas while preserving stability in static regions, along with a dedicated constraint to enforce lighting–material disentanglement. To facilitate evaluation, we construct the first synthetic benchmark containing both static and dynamic content under multiple lighting conditions. Experiments demonstrate that our method improves albedo estimation by 23% in LPIPS and achieves a 15% performance gain in relighting tasks, significantly outperforming current state-of-the-art approaches.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Motion and Path Planning
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In 3D reconstruction, the problem of inverse rendering, namely recovering the illumination of the scene and the material properties, is fundamental. Existing Gaussian Splatting-based methods primarily target static scenes and often assume simplified or moderate lighting to avoid entangling shadows with surface appearance. This limits their ability to accurately separate lighting effects from material properties, particularly in real-world conditions. We address this limitation by leveraging dynamic elements - regions of the scene that undergo motion - as a supervisory signal for inverse rendering. Motion reveals the same surfaces under varying lighting conditions, providing stronger cues for disentangling material and illumination. This thesis is supported by our experimental results which show we improve LPIPS by 23% for albedo estimation and by 15% for scene relighting relative to next-best baseline. To this end, we introduce LumiMotion, the first Gaussian-based approach that leverages dynamics for inverse rendering and operates in arbitrary dynamic scenes. Our method learns a dynamic 2D Gaussian Splatting representation that employs a set of novel constraints which encourage the dynamic regions of the scene to deform, while keeping static regions stable. As we demonstrate, this separation is crucial for correct optimization of the albedo. Finally, we release a new synthetic benchmark comprising five scenes under four lighting conditions, each in both static and dynamic variants, for the first time enabling systematic evaluation of inverse rendering methods in dynamic environments and challenging lighting. Link to project page: https://joaxkal.github.io/LumiMotion/

Problem

• inverse rendering
• dynamic scenes
• Gaussian Splatting
• lighting estimation
• material disentanglement

Innovation

• Gaussian Splatting
• inverse rendering
• scene dynamics
• relighting
• albedo estimation