UIP2P: Unsupervised Instruction-based Image Editing via Cycle Edit Consistency

2024-12-19

arXiv.org

AI Summary

Existing instruction-driven image editing methods heavily rely on manually annotated or model-generated image–instruction–edited-image triplets, leading to high annotation costs, poor generalization, and error accumulation. To address this, we propose the first unsupervised instruction editing framework that requires no ground-truth edited images—only image–text pairs or weakly labeled triplets suffice for training. Our core innovation is the Cycle Editing Consistency (CEC) mechanism, which jointly enforces consistency between forward editing and backward reconstruction in both pixel and attention spaces. CEC integrates bidirectional diffusion modeling with cross-space attention alignment to ensure structural and semantic coherence. Experiments demonstrate that our method outperforms supervised baselines across diverse editing tasks, significantly improving edit fidelity and instruction adherence while eliminating dependence on high-quality annotated data.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We propose an unsupervised model for instruction-based image editing that eliminates the need for ground-truth edited images during training. Existing supervised methods depend on datasets containing triplets of input image, edited image, and edit instruction. These are generated by either existing editing methods or human-annotations, which introduce biases and limit their generalization ability. Our method addresses these challenges by introducing a novel editing mechanism called Cycle Edit Consistency (CEC), which applies forward and backward edits in one training step and enforces consistency in image and attention spaces. This allows us to bypass the need for ground-truth edited images and unlock training for the first time on datasets comprising either real image-caption pairs or image-caption-edit triplets. We empirically show that our unsupervised technique performs better across a broader range of edits with high fidelity and precision. By eliminating the need for pre-existing datasets of triplets, reducing biases associated with supervised methods, and proposing CEC, our work represents a significant advancement in unblocking scaling of instruction-based image editing.

Problem

• Eliminating need for ground-truth edited images during training
• Reducing biases from existing editing methods and human annotations
• Enabling training on real image-caption pairs without triplets

Innovation

• Unsupervised training without ground-truth edited images
• Edit Reversibility Constraint with forward and reverse edits
• Alignment enforcement across image, text, and attention spaces