RGS-DR: Reflective Gaussian Surfels with Deferred Rendering for Shiny Objects

2025-04-25

AI Summary

Existing inverse rendering methods (e.g., NeRF, 3D Gaussian Splatting) struggle to accurately model view-dependent appearance of specular objects, leading to reconstruction artifacts, blurred reflections, and limited support for relighting and editing. To address this, we propose RGS-DR: a geometry-material joint explicit representation built upon 2D Gaussian surfels. We introduce a novel multi-level cubic mipmap approximation for environment map integration and a spherical mipmap-based directional encoding residual pathway, significantly improving reflection fidelity. Coupled with a differentiable deferred shading pipeline, RGS-DR enables efficient, high-fidelity rendering. Experiments demonstrate that RGS-DR outperforms state-of-the-art methods on reconstruction and rendering of glossy and reflective objects—effectively suppressing artifacts, preserving sharp specular highlights, and enabling high-quality relighting and scene editing.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce RGS-DR, a novel inverse rendering method for reconstructing and rendering glossy and reflective objects with support for flexible relighting and scene editing. Unlike existing methods (e.g., NeRF and 3D Gaussian Splatting), which struggle with view-dependent effects, RGS-DR utilizes a 2D Gaussian surfel representation to accurately estimate geometry and surface normals, an essential property for high-quality inverse rendering. Our approach explicitly models geometric and material properties through learnable primitives rasterized into a deferred shading pipeline, effectively reducing rendering artifacts and preserving sharp reflections. By employing a multi-level cube mipmap, RGS-DR accurately approximates environment lighting integrals, facilitating high-quality reconstruction and relighting. A residual pass with spherical-mipmap-based directional encoding further refines the appearance modeling. Experiments demonstrate that RGS-DR achieves high-quality reconstruction and rendering quality for shiny objects, often outperforming reconstruction-exclusive state-of-the-art methods incapable of relighting.

Problem

• Accurate reconstruction of glossy and reflective objects
• Handling view-dependent effects in inverse rendering
• High-quality relighting and scene editing capabilities

Innovation

• Uses 2D Gaussian surfels for geometry estimation
• Employs deferred shading pipeline for rendering
• Utilizes multi-level cube mipmap for lighting