PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions

2024-09-23

arXiv.org

AI Summary

This paper introduces OmniControl, a unified image–text–image generation framework addressing the challenge of jointly modeling heterogeneous vision-language tasks—such as instruction-driven image generation, editing, inpainting, and cross-domain translation—in multimodal assistants. Methodologically, we construct the first Omni Pixel-to-Pixel instruction-tuning dataset; propose a dynamic arbitrary-resolution mechanism and a dual-aware (structural and semantic) guidance strategy; and design a resolution-adaptive Diffusion Transformer (DiT) architecture that fuses language templates with multimodal features. Experiments demonstrate that OmniControl significantly outperforms existing methods on multi-resolution generation and comprehension benchmarks. Moreover, it exhibits strong generalization to unseen tasks and real-world user instructions, achieving high alignment with human perceptual judgments.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions. The code and related resources are available at https://github.com/AFeng-x/PixWizard

Problem

• Unified image-text-to-image generation framework
• Dynamic image processing by aspect ratio
• Structure and semantic-aware image information fusion

Innovation

• Unified image-text-to-image framework
• Diffusion Transformers with flexible resolution
• Structure-aware and semantic-aware guidance mechanisms