PhyGaP: Physically-Grounded Gaussians with Polarization Cues

2026-03-14

AI Summary

Existing 3D Gaussian splatting methods struggle to accurately reconstruct the physical appearance properties of reflective objects—such as albedo and reflectance—leading to distorted relighting results. This work addresses this limitation by introducing polarimetric cues into the 3D Gaussian splatting framework for the first time, proposing Polarization-aware Deferred Rendering (PolarDR) and a self-occlusion-aware environment map (GridMap) to enable physically plausible reflectance decomposition and consistent relighting. By moving beyond the constraints of RGB-only inverse rendering, the method achieves significant improvements over state-of-the-art approaches on both synthetic and real-world scenes, yielding approximately 2 dB higher PSNR and a 45.7% reduction in normal cosine distance error, thereby enabling high-quality, physics-based inverse rendering and relighting.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have demonstrated great success in modeling reflective 3D objects and their interaction with the environment via deferred rendering (DR). However, existing methods often struggle with correctly reconstructing physical attributes such as albedo and reflectance, and therefore they do not support high-fidelity relighting. Observing that this limitation stems from the lack of shape and material information in RGB images, we present PhyGaP, a physically-grounded 3DGS method that leverages polarization cues to facilitate precise reflection decomposition and visually consistent relighting of reconstructed objects. Specifically, we design a polarimetric deferred rendering (PolarDR) process to model polarization by reflection, and a self-occlusion-aware environment map building technique (GridMap) to resolve indirect lighting of non-convex objects. We validate on multiple synthetic and real-world scenes, including those featuring only partial polarization cues, that PhyGaP not only excels in reconstructing the appearance and surface normal of reflective 3D objects (~2 dB in PSNR and 45.7% in Cosine Distance better than existing RGB-based methods on average), but also achieves state-of-the-art inverse rendering and relighting capability. Our code will be released soon.

Problem

• 3D Gaussian Splatting
• relighting
• reflective objects
• physical attributes
• polarization cues

Innovation

• Polarization
• 3D Gaussian Splatting
• Physically-based Rendering
• Relighting
• Inverse Rendering