Overpainting: Localized Context-aware Diffusion Image Editing

2026-09-09

AI Summary

本文提出了一种名为'overpainting'的图像编辑方法，通过自适应预训练模型和自动训练数据生成管道，实现了对图像局部区域的精确或宽松编辑。

Abstract

We present "overpainting", an image editing operation which offers both control over the location of the edit and awareness of the previous content in that location. The overpainted area is given by a trimap, where white-annotated pixels must be edited, gray-annotated pixels may be edited, and black-annotated pixels must not be edited. This enables both precise and loose control, depending on user intent. We implement overpainting by adapting a pretrained image editing diffusion model using a combination of joint attention and low-rank adaption across input images with attention-dropout to balance the information flow between noise, source and mask images. We present a novel, automated, training data generation pipeline that (1) generates a set of candidate image pairs leveraging existing language-based editing models, (2) carefully curates those pairs, and (3) extracts a trimap from each usable pair. We demonstrate the versatility of our overpainting model on a wide range of editing tasks.

Problem

• image editing
• context-aware
• localized control
• trimap

Innovation

• overpainting
• trimap
• joint attention
• low-rank adaption
• attention-dropout