GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editing

2024-12-03

arXiv.org

AI Summary

To address the limited generalization of conventional data augmentation methods under large source-target domain discrepancies in domain adaptation, this paper proposes GenMix—a prompt-guided generative data augmentation framework. GenMix leverages diffusion models for controllable image editing, integrating conditional prompt engineering with fractal mask fusion to simultaneously enhance intra-domain discriminability and cross-domain robustness while preserving semantic consistency and label fidelity. Its core innovation lies in the first synergistic integration of prompt-driven controllable editing and fractal mixing for data augmentation. Extensive experiments across eight public benchmarks demonstrate that GenMix consistently outperforms state-of-the-art methods, achieving significant improvements in cross-domain classification, fine-grained recognition, self-supervised pretraining, few-shot learning, and adversarial robustness tasks.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Data augmentation is widely used to enhance generalization in visual classification tasks. However, traditional methods struggle when source and target domains differ, as in domain adaptation, due to their inability to address domain gaps. This paper introduces GenMix, a generalizable prompt-guided generative data augmentation approach that enhances both in-domain and cross-domain image classification. Our technique leverages image editing to generate augmented images based on custom conditional prompts, designed specifically for each problem type. By blending portions of the input image with its edited generative counterpart and incorporating fractal patterns, our approach mitigates unrealistic images and label ambiguity, improving the performance and adversarial robustness of the resulting models. Efficacy of our method is established with extensive experiments on eight public datasets for general and fine-grained classification, in both in-domain and cross-domain settings. Additionally, we demonstrate performance improvements for self-supervised learning, learning with data scarcity, and adversarial robustness. As compared to the existing state-of-the-art methods, our technique achieves stronger performance across the board.

Problem

• Enhancing image classification across domain gaps
• Improving adversarial robustness with generative data augmentation
• Addressing data scarcity in visual classification tasks

Innovation

• Leverages image editing with custom prompts
• Blends input images with generative counterparts
• Incorporates fractal patterns to enhance realism