CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Model

2025-04-11

AI Summary

This paper addresses the lack of controllability and editability in sketch generation for artistic creation. Methodologically, it proposes a controllable, progressive text-to-sketch generation framework: (1) It introduces the first binary sketch representation based on Unsigned Distance Fields (UDFs), enhancing diffusion model stability for generating sketches with crisp, well-defined edges; (2) It constructs the first large-scale, high-quality text-sketch paired dataset; and (3) It designs a multi-round refinement architecture that incorporates human editing feedback, enabling a closed-loop generation pipeline—from text or bounding-box prompts to coarse sketches, interactive editing, and finally to high-fidelity binary sketches. Experiments demonstrate significant improvements over state-of-the-art baselines in semantic alignment, structural clarity, and user controllability, effectively supporting artist-in-the-loop generative workflows.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Sketches serve as fundamental blueprints in artistic creation because sketch editing is easier and more intuitive than pixel-level RGB image editing for painting artists, yet sketch generation remains unexplored despite advancements in generative models. We propose a novel framework CoProSketch, providing prominent controllability and details for sketch generation with diffusion models. A straightforward method is fine-tuning a pretrained image generation diffusion model with binarized sketch images. However, we find that the diffusion models fail to generate clear binary images, which makes the produced sketches chaotic. We thus propose to represent the sketches by unsigned distance field (UDF), which is continuous and can be easily decoded to sketches through a lightweight network. With CoProSketch, users generate a rough sketch from a bounding box and a text prompt. The rough sketch can be manually edited and fed back into the model for iterative refinement and will be decoded to a detailed sketch as the final result. Additionally, we curate the first large-scale text-sketch paired dataset as the training data. Experiments demonstrate superior semantic consistency and controllability over baselines, offering a practical solution for integrating user feedback into generative workflows.

Problem

• Generating clear binary sketches using diffusion models
• Providing controllability and iterative refinement for sketch generation
• Creating a large-scale text-sketch dataset for training

Innovation

• UDF representation for continuous sketch decoding
• Iterative refinement with user feedback
• Large-scale text-sketch paired dataset