Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections

2024-09-23

arXiv.org

AI Summary

This work addresses the challenge of generating geometrically consistent, photorealistic, and user-controllable specular reflections—long-standing limitations of diffusion models. We formulate the task as depth-conditioned image inpainting, enabling users to freely specify the position and pose of mirror surfaces. Our approach is the first to achieve controllable, geometry-aware (depth- and normal-aligned), and color-faithful reflection synthesis for scene objects. To support learning-based reflection generation, we introduce SynMirror, the first large-scale synthetic specular reflection dataset comprising 198K high-fidelity samples. We further propose MirrorFusion, a diffusion-based multimodal geometric supervision framework that jointly leverages depth, surface normals, and semantic priors for reflection inpainting. Extensive quantitative and qualitative evaluations on SynMirror demonstrate that MirrorFusion significantly outperforms existing methods in reflection realism, shape consistency, and appearance fidelity.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198k samples rendered from 66k unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality, realistic, shape and appearance-aware reflections of real-world objects. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion-based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike. The project page is available at: https://val.cds.iisc.ac.in/reflecting-reality.github.io/.

Problem

• Realistic Reflections
• Virtual Environment
• Complex Models

Innovation

• MirrorFusion
• SynMirror
• Controlled Reflection Generation