OmniSR: Shadow Removal under Direct and Indirect Lighting

2024-10-02

arXiv.org

AI Summary

This work addresses the challenging problem of shadow removal under coexisting direct and indirect illumination—particularly focusing on residual shadows caused by occlusion of indirect light, and the weak-supervision challenge arising from the absence of paired shadow-free images. To tackle these issues, we propose: (1) the first differentiable synthetic rendering pipeline capable of modeling both direct and indirect illumination; (2) DSID, a large-scale dataset comprising over 30,000 paired shadow/shadow-free images with dual-illumination annotations; and (3) a cross-modal attention network integrating semantic segmentation (SegFormer) and geometric depth priors. Our method achieves significant improvements over state-of-the-art approaches across multiple benchmarks and real-world indoor/outdoor scenes, with average gains of 2.1 dB in PSNR and 0.023 in SSIM. It demonstrates strong generalization capability and consistently enhances performance of downstream vision tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Shadows can originate from occlusions in both direct and indirect illumination. Although most current shadow removal research focuses on shadows caused by direct illumination, shadows from indirect illumination are often just as pervasive, particularly in indoor scenes. A significant challenge in removing shadows from indirect illumination is obtaining shadow-free images to train the shadow removal network. To overcome this challenge, we propose a novel rendering pipeline for generating shadowed and shadow-free images under direct and indirect illumination, and create a comprehensive synthetic dataset that contains over 30,000 image pairs, covering various object types and lighting conditions. We also propose an innovative shadow removal network that explicitly integrates semantic and geometric priors through concatenation and attention mechanisms. The experiments show that our method outperforms state-of-the-art shadow removal techniques and can effectively generalize to indoor and outdoor scenes under various lighting conditions, enhancing the overall effectiveness and applicability of shadow removal methods.

Problem

• Shadow Removal
• Reflection Occlusion
• No Shadow-Free Reference

Innovation

• OmniSR
• Shadow Removal
• Large-scale Dataset