OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting

2025-03-11

AI Summary

In object-centric image editing, object removal and insertion have long been hindered by the complexity of physical interaction modeling, scarcity of paired training data, and geometric-physical inconsistency. This paper proposes a decoupled joint modeling paradigm for insertion and removal, treating them as interdependent, invertible processes to achieve geometrically consistent and physically plausible high-fidelity editing. Methodologically, we leverage pretrained diffusion priors and integrate paired coarse optimization with a CycleFlow-driven unpaired fine-tuning framework. We further introduce the first reference-free CFD (Consistency, Fidelity, and Detection) evaluation metric, enabling quantitative assessment of contextual consistency and hallucination. Our approach achieves state-of-the-art performance across multiple benchmarks: significantly improved foreground removal accuracy, more natural object insertion blending, and superior preservation of scene geometry and material attributes.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Diffusion-based generative models have revolutionized object-oriented image editing, yet their deployment in realistic object removal and insertion remains hampered by challenges such as the intricate interplay of physical effects and insufficient paired training data. In this work, we introduce OmniPaint, a unified framework that re-conceptualizes object removal and insertion as interdependent processes rather than isolated tasks. Leveraging a pre-trained diffusion prior along with a progressive training pipeline comprising initial paired sample optimization and subsequent large-scale unpaired refinement via CycleFlow, OmniPaint achieves precise foreground elimination and seamless object insertion while faithfully preserving scene geometry and intrinsic properties. Furthermore, our novel CFD metric offers a robust, reference-free evaluation of context consistency and object hallucination, establishing a new benchmark for high-fidelity image editing. Project page: https://yeates.github.io/OmniPaint-Page/

Problem

• Challenges in realistic object removal and insertion in images.
• Insufficient paired training data for diffusion-based generative models.
• Need for preserving scene geometry and intrinsic properties during editing.

Innovation

• Unified framework for object removal and insertion
• Pre-trained diffusion prior with progressive training
• CFD metric for reference-free image editing evaluation