TranSplat: Surface Embedding-guided 3D Gaussian Splatting for Transparent Object Manipulation

2025-02-11

AI Summary

Transparent objects impede depth sensing due to their non-Lambertian optical properties, resulting in severe depth missing and inaccurate 3D reconstruction; existing methods struggle to model such reflectance characteristics while ensuring inter-frame consistency. This paper proposes a surface-embedding-guided 3D Gaussian rasterization framework for depth completion. We introduce the first latent diffusion model (LDM)-based approach to generate continuous, geometrically consistent surface embeddings that explicitly encode non-Lambertian reflectance of transparent objects. Our method jointly optimizes RGB-guided surface embedding learning, 3D Gaussian rasterization rendering, and depth completion. Evaluated on synthetic and real-world transparent-object datasets—as well as robotic grasping tasks—it achieves high-fidelity, dense, and temporally coherent depth reconstruction. To foster reproducibility and further research, we release our code, pretrained models, and a newly constructed transparent-object benchmark dataset.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transparent, Interpretable, Explainable ML
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Transparent object manipulation remains a sig- nificant challenge in robotics due to the difficulty of acquiring accurate and dense depth measurements. Conventional depth sensors often fail with transparent objects, resulting in in- complete or erroneous depth data. Existing depth completion methods struggle with interframe consistency and incorrectly model transparent objects as Lambertian surfaces, leading to poor depth reconstruction. To address these challenges, we propose TranSplat, a surface embedding-guided 3D Gaussian Splatting method tailored for transparent objects. TranSplat uses a latent diffusion model to generate surface embeddings that provide consistent and continuous representations, making it robust to changes in viewpoint and lighting. By integrating these surface embeddings with input RGB images, TranSplat effectively captures the complexities of transparent surfaces, enhancing the splatting of 3D Gaussians and improving depth completion. Evaluations on synthetic and real-world transpar- ent object benchmarks, as well as robot grasping tasks, show that TranSplat achieves accurate and dense depth completion, demonstrating its effectiveness in practical applications. We open-source synthetic dataset and model: https://github. com/jeongyun0609/TranSplat

Problem

• Accurate depth measurement for transparent objects
• Interframe consistency in depth completion
• Modeling transparent surfaces beyond Lambertian assumptions

Innovation

• Surface embedding-guided 3D Gaussian Splatting
• Latent diffusion model for consistent embeddings
• RGB integration for transparent surface complexity