Beyond Imperfections: A Conditional Inpainting Approach for End-to-End Artifact Removal in VTON and Pose Transfer

2024-10-05

arXiv.org

AI Summary

To address pervasive visual artifacts—such as distortion, misalignment, and blurring—in virtual try-on (VTON) and pose transfer, this paper proposes the first end-to-end conditional inpainting framework for automatic artifact detection and high-fidelity removal. Methodologically, it integrates semantic-guided mask prediction, a multi-scale feature fusion inpainting network, and a conditional diffusion model to jointly reconstruct corrupted regions with structural and textural coherence. Key contributions include: (1) the first fine-grained mask-annotated dataset specifically designed for artifact removal in VTON and pose transfer; and (2) the first end-to-end trainable inpainting architecture tailored to this task. Extensive experiments demonstrate state-of-the-art performance in quantitative metrics (PSNR, LPIPS) and user studies, significantly improving image fidelity and perceptual naturalness over prior methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

Artifacts often degrade the visual quality of virtual try-on (VTON) and pose transfer applications, impacting user experience. This study introduces a novel conditional inpainting technique designed to detect and remove such distortions, improving image aesthetics. Our work is the first to present an end-to-end framework addressing this specific issue, and we developed a specialized dataset of artifacts in VTON and pose transfer tasks, complete with masks highlighting the affected areas. Experimental results show that our method not only effectively removes artifacts but also significantly enhances the visual quality of the final images, setting a new benchmark in computer vision and image processing.

Problem

• Virtual Try-On (VTON)
• Pose Transfer
• Image Flaw Removal

Innovation

• Virtual Try-On
• Flaw Detection and Correction
• Computer Vision Advancement