SoftShadow: Leveraging Penumbra-Aware Soft Masks for Shadow Removal

2024-09-11

arXiv.org

AI Summary

Existing shadow removal methods rely on binary shadow masks, often introducing artifacts near shadow boundaries. To address this, we propose an end-to-end soft shadow modeling framework that, for the first time, incorporates the physical formation mechanism of penumbra into deep learning. Leveraging a pre-trained Segment Anything Model (SAM) as structural prior, our method jointly models both umbra and penumbra regions to generate continuous, physically consistent soft shadow masks. We further introduce a penumbra-aware constraint loss, co-optimized with the shadow removal objective. This design effectively alleviates boundary distortion and preserves fine-grained shadow structure. Extensive experiments demonstrate state-of-the-art performance on mainstream benchmarks including ISTD and SBU, with improved generalization across diverse shadow scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Life-Long and Continual Learning
• Intelligent Robots: Manipulation

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in deep learning have yielded promising results for the image shadow removal task. However, most existing methods rely on binary pre-generated shadow masks. The binary nature of such masks could potentially lead to artifacts near the boundary between shadow and non-shadow areas. In view of this, inspired by the physical model of shadow formation, we introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a extit{SoftShadow} framework by leveraging the prior knowledge of pretrained SAM and integrating physical constraints. Specifically, we jointly tune the SAM and the subsequent shadow removal network using penumbra formation constraint loss and shadow removal loss. This framework enables accurate predictions of penumbra (partially shaded regions) and umbra (fully shaded regions) areas while simultaneously facilitating end-to-end shadow removal. Through extensive experiments on popular datasets, we found that our SoftShadow framework, which generates soft masks, can better restore boundary artifacts, achieve state-of-the-art performance, and demonstrate superior generalizability.

Problem

• Addresses artifacts near shadow boundaries using soft masks.
• Proposes SoftShadow framework for penumbra-aware shadow removal.
• Improves shadow removal accuracy and generalizability with soft masks.

Innovation

• Introduces soft shadow masks for artifact-free shadow removal.
• Integrates SAM with physical constraints for accurate penumbra prediction.
• Uses joint tuning with multiple losses for end-to-end shadow removal.