Refracting Reality: Generating Images with Realistic Transparent Objects

2025-11-21

AI Summary

Existing generative models struggle to accurately model the optical behavior of transparent objects, particularly failing to adhere to Snell’s law in refraction rendering. This work introduces the first diffusion-based generative framework that explicitly incorporates geometric optics principles: at each denoising step, it computes and applies pixel-wise spatial warping and ray-path fusion grounded in Snell’s law, conditioned on object geometry and material parameters. Concurrently, it synthesizes a center-aligned panoramic auxiliary view to physically consistently reconstruct occluded surface appearances—including reflection, refraction, absorption, and scattering. The method requires no additional annotations or pretraining. It significantly improves optical plausibility and visual realism of transparent-object images, outperforming state-of-the-art methods in both qualitative and quantitative evaluations.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Non-convex Optimization
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Algorithmic accountability and transparency on the web
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Generative image models can produce convincingly real images, with plausible shapes, textures, layouts and lighting. However, one domain in which they perform notably poorly is in the synthesis of transparent objects, which exhibit refraction, reflection, absorption and scattering. Refraction is a particular challenge, because refracted pixel rays often intersect with surfaces observed in other parts of the image, providing a constraint on the color. It is clear from inspection that generative models have not distilled the laws of optics sufficiently well to accurately render refractive objects. In this work, we consider the problem of generating images with accurate refraction, given a text prompt. We synchronize the pixels within the object's boundary with those outside by warping and merging the pixels using Snell's Law of Refraction, at each step of the generation trajectory. For those surfaces that are not directly observed in the image, but are visible via refraction or reflection, we recover their appearance by synchronizing the image with a second generated image -- a panorama centered at the object -- using the same warping and merging procedure. We demonstrate that our approach generates much more optically-plausible images that respect the physical constraints.

Problem

• Generative models struggle with realistic transparent object synthesis
• Refraction poses challenges due to optical law constraints
• Method improves image generation with accurate refractive effects

Innovation

• Using Snell's Law to warp pixels during generation
• Synchronizing object boundaries with external pixels
• Merging images with panoramas for hidden surfaces