Generating Sketches in a Hierarchical Auto-Regressive Process for Flexible Sketch Drawing Manipulation at Stroke-Level

2025-11-11

AI Summary

To address the challenge of dynamically adjusting strokes during sketch generation, this paper proposes a hierarchical autoregressive sketch generation framework. The method decomposes generation into three stages: stroke semantic embedding prediction, spatial anchor localization, and sequential drawing action decoding. By employing autoregressive modeling, each stroke generation step explicitly conditions on historical states, enabling real-time, stroke-level editing and conditional intervention during synthesis. To our knowledge, this is the first model achieving dynamic, stroke-level manipulation *during* generation. Experiments demonstrate that the framework maintains high visual fidelity while significantly improving structural plausibility and interactive controllability, outperforming state-of-the-art sketch generation methods across multiple benchmarks.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Generating sketches with specific patterns as expected, i.e., manipulating sketches in a controllable way, is a popular task. Recent studies control sketch features at stroke-level by editing values of stroke embeddings as conditions. However, in order to provide generator a global view about what a sketch is going to be drawn, all these edited conditions should be collected and fed into generator simultaneously before generation starts, i.e., no further manipulation is allowed during sketch generating process. In order to realize sketch drawing manipulation more flexibly, we propose a hierarchical auto-regressive sketch generating process. Instead of generating an entire sketch at once, each stroke in a sketch is generated in a three-staged hierarchy: 1) predicting a stroke embedding to represent which stroke is going to be drawn, and 2) anchoring the predicted stroke on the canvas, and 3) translating the embedding to a sequence of drawing actions to form the full sketch. Moreover, the stroke prediction, anchoring and translation are proceeded auto-regressively, i.e., both the recently generated strokes and their positions are considered to predict the current one, guiding model to produce an appropriate stroke at a suitable position to benefit the full sketch generation. It is flexible to manipulate stroke-level sketch drawing at any time during generation by adjusting the exposed editable stroke embeddings.

Problem

• Enabling flexible stroke-level manipulation during sketch generation process
• Overcoming limitations of pre-set stroke conditions before generation starts
• Developing hierarchical auto-regressive process for dynamic stroke prediction and placement

Innovation

• Hierarchical auto-regressive process for sketch generation
• Stroke-level manipulation via editable embeddings
• Three-staged hierarchy: predict, anchor, translate strokes