DreamOmni3: Scribble-based Editing and Generation

2025-12-27

AI Summary

Existing text-guided image editing and generation models struggle with precise spatial localization of edit regions and faithful modeling of fine-grained visual details. To address this, we propose a sketch-driven unified editing and generation framework that enables flexible user control via hand-drawn scribbles, textual instructions, and reference images within a GUI interface. Methodologically, we introduce a novel dual-image joint encoding scheme—simultaneously encoding the original image and the scribble map—augmented with color-coded region identifiers, shared index embeddings, and positional encodings. We further construct the first comprehensive benchmark and synthetic dataset for scribble-based editing/generation, covering seven fine-grained tasks and built upon DreamOmni2. Extensive experiments demonstrate state-of-the-art performance across diverse editing and generation tasks. Our code, pre-trained models, and benchmark are fully open-sourced.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web crawling and indexing
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recently unified generation and editing models have achieved remarkable success with their impressive performance. These models rely mainly on text prompts for instruction-based editing and generation, but language often fails to capture users intended edit locations and fine-grained visual details. To this end, we propose two tasks: scribble-based editing and generation, that enables more flexible creation on graphical user interface (GUI) combining user textual, images, and freehand sketches. We introduce DreamOmni3, tackling two challenges: data creation and framework design. Our data synthesis pipeline includes two parts: scribble-based editing and generation. For scribble-based editing, we define four tasks: scribble and instruction-based editing, scribble and multimodal instruction-based editing, image fusion, and doodle editing. Based on DreamOmni2 dataset, we extract editable regions and overlay hand-drawn boxes, circles, doodles or cropped image to construct training data. For scribble-based generation, we define three tasks: scribble and instruction-based generation, scribble and multimodal instruction-based generation, and doodle generation, following similar data creation pipelines. For the framework, instead of using binary masks, which struggle with complex edits involving multiple scribbles, images, and instructions, we propose a joint input scheme that feeds both the original and scribbled source images into the model, using different colors to distinguish regions and simplify processing. By applying the same index and position encodings to both images, the model can precisely localize scribbled regions while maintaining accurate editing. Finally, we establish comprehensive benchmarks for these tasks to promote further research. Experimental results demonstrate that DreamOmni3 achieves outstanding performance, and models and code will be publicly released.

Problem

• Enables flexible image editing using scribbles and text
• Creates data for scribble-based editing and generation tasks
• Proposes joint input scheme for precise scribble localization

Innovation

• Joint input scheme with original and scribbled images
• Data synthesis pipeline for scribble-based editing and generation
• Color differentiation and encoding for precise region localization