Removing Reflections from RAW Photos

2024-02-28

arXiv.org

AI Summary

This work addresses real-world reflection removal in the RAW domain for consumer photography. We propose the first end-to-end RAW-domain reflection elimination method, comprising a two-stage architecture: a 256p base CNN (with dual inputs—primary-camera RAW image and an optional reverse-context image from a front-facing camera) followed by a lightweight super-resolution network. The model is trained exclusively on photometrically and geometrically consistent synthetic RAW reflection data. Key contributions include: (i) the first end-to-end RAW-domain reflection decomposition; (ii) incorporation of reverse-context images to enhance reflection modeling; and (iii) state-of-the-art performance achieved using synthetic data alone—surpassing prior methods trained on real data. Our approach achieves superior results on a real-world RAW test set and processes 1K previews in 4.5–6.5 seconds on MacBook and iPhone 14 Pro devices.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Hardware-aware ML
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Data transparency and provenance

Abstract

We describe a system to remove real-world reflections from images for consumer photography. Our system operates on linear (RAW) photos, and accepts an optional contextual photo looking in the opposite direction (e.g., the"selfie"camera on a mobile device). This optional photo disambiguates what should be considered the reflection. The system is trained solely on synthetic mixtures of real RAW photos, which we combine using a reflection simulation that is photometrically and geometrically accurate. Our system comprises a base model that accepts the captured photo and optional context photo as input, and runs at 256p, followed by an up-sampling model that transforms 256p images to full resolution. The system produces preview images at 1K in 4.5-6.5s on a MacBook or iPhone 14 Pro. We show SOTA results on RAW photos that were captured in the field to embody typical consumer photos, and show that training on RAW simulation data improves performance more than the architectural variations among prior works.

Problem

• Remove reflections from RAW photos for consumer photography
• Use optional contextual photo to disambiguate reflections
• Train system on synthetic RAW photo mixtures

Innovation

• Uses RAW photos and optional contextual photo
• Trained on synthetic mixtures of real RAW photos
• Combines base model and up-sampling model