Region-to-Region: Enhancing Generative Image Harmonization with Adaptive Regional Injection

2025-08-13

AI Summary

Image harmonization aims to achieve visual consistency between foreground and background in composite images. However, existing latent diffusion model (LDM)-based methods suffer from inadequate high-frequency detail preservation, insufficient local coherence, and inaccurate physical lighting modeling; moreover, prevailing synthetic datasets lack diversity and photorealistic plausibility. To address these limitations, we propose a region-to-region transformation mechanism and a Mask-Aware Channel Attention (MACA) module, design the Clear-VAE encoder for enhanced high-frequency detail retention, and introduce RPHarmony—a novel dataset featuring rich local variations and physically grounded illumination. Our method integrates adaptive filtering, mask-aware channel attention, and stochastic Poisson blending for dynamic foreground harmonization. Experiments demonstrate significant improvements over state-of-the-art methods in both quantitative metrics (e.g., LPIPS, FID) and perceptual quality. Ablations confirm RPHarmony’s effectiveness in enhancing model generalization. Code, models, and data are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

The goal of image harmonization is to adjust the foreground in a composite image to achieve visual consistency with the background. Recently, latent diffusion model (LDM) are applied for harmonization, achieving remarkable results. However, LDM-based harmonization faces challenges in detail preservation and limited harmonization ability. Additionally, current synthetic datasets rely on color transfer, which lacks local variations and fails to capture complex real-world lighting conditions. To enhance harmonization capabilities, we propose the Region-to-Region transformation. By injecting information from appropriate regions into the foreground, this approach preserves original details while achieving image harmonization or, conversely, generating new composite data. From this perspective, We propose a novel model R2R. Specifically, we design Clear-VAE to preserve high-frequency details in the foreground using Adaptive Filter while eliminating disharmonious elements. To further enhance harmonization, we introduce the Harmony Controller with Mask-aware Adaptive Channel Attention (MACA), which dynamically adjusts the foreground based on the channel importance of both foreground and background regions. To address the limitation of existing datasets, we propose Random Poisson Blending, which transfers color and lighting information from a suitable region to the foreground, thereby generating more diverse and challenging synthetic images. Using this method, we construct a new synthetic dataset, RPHarmony. Experiments demonstrate the superiority of our method over other methods in both quantitative metrics and visual harmony. Moreover, our dataset helps the model generate more realistic images in real examples. Our code, dataset, and model weights have all been released for open access.

Problem

• Enhance image harmonization with adaptive regional injection
• Preserve foreground details while achieving visual consistency
• Generate diverse synthetic data for realistic harmonization

Innovation

• Region-to-Region transformation for adaptive regional injection
• Clear-VAE with Adaptive Filter preserves high-frequency details
• Harmony Controller uses Mask-aware Adaptive Channel Attention