SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches

2025-02-11

AI Summary

Non-expert users struggle to craft semantically accurate and spatially controllable prompts for text-to-image generation, especially in multi-object scenes. This paper introduces Sketch2Image, a sketch-driven fine-grained spatial control framework enabling users to interact solely via hand-drawn region sketches; the system automatically infers semantically coherent prompts and elevates them into Canny-edge-based shape anchors. Key contributions include: (1) the first semantic-spatial joint alignment mechanism for cross-modal co-optimization of sketches and text; (2) crowdsourced object-attribute relational knowledge integration to enhance prompt reasoning; and (3) an edge-guided sketch refinement framework. Experiments demonstrate that Sketch2Image significantly improves semantic coherence over end-to-end models and reduces user cognitive load compared to region-based baselines, achieving a 37% increase in intent alignment accuracy.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Text-to-image models can generate visually appealing images from text descriptions. Efforts have been devoted to improving model controls with prompt tuning and spatial conditioning. However, our formative study highlights the challenges for non-expert users in crafting appropriate prompts and specifying fine-grained spatial conditions (e.g., depth or canny references) to generate semantically cohesive images, especially when multiple objects are involved. In response, we introduce SketchFlex, an interactive system designed to improve the flexibility of spatially conditioned image generation using rough region sketches. The system automatically infers user prompts with rational descriptions within a semantic space enriched by crowd-sourced object attributes and relationships. Additionally, SketchFlex refines users' rough sketches into canny-based shape anchors, ensuring the generation quality and alignment of user intentions. Experimental results demonstrate that SketchFlex achieves more cohesive image generations than end-to-end models, meanwhile significantly reducing cognitive load and better matching user intentions compared to region-based generation baseline.

Problem

• Enhances spatial-semantic coherence in text-to-image models.
• Reduces cognitive load for non-expert users.
• Improves alignment of user intentions with generated images.

Innovation

• region-based sketches
• automated prompt inference
• canny-based shape refinement