AutoSketch: VLM-assisted Style-Aware Vector Sketch Completion

2025-02-07

AI Summary

Existing sketch generation methods support only from-scratch synthesis, making partial sketch completion while preserving users’ original hand-drawn styles highly challenging. AutoSketch introduces the first vector-based framework for partial sketch completion in complex scenes, enabling semantic- and style-aligned completion guided by natural language prompts (e.g., “a man and woman conversing in a park”). Its core innovations include: (1) leveraging a pre-trained vision-language model (VLM) to extract natural language descriptions of sketch style and construct a diffusion prior; (2) generating executable code for fine-grained style adaptation; and (3) integrating text-enhanced diffusion optimization with vectorized stroke parameterization. On multi-style sketch datasets, AutoSketch achieves a 12.6% improvement in quantitative metrics, supports interactive editing, and generalizes across diverse styles. Ablation studies and user studies confirm its effectiveness.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The ability to automatically complete a partial sketch that depicts a complex scene, e.g.,"a woman chatting with a man in the park", is very useful. However, existing sketch generation methods create sketches from scratch; they do not complete a partial sketch in the style of the original. To address this challenge, we introduce AutoSketch, a styleaware vector sketch completion method that accommodates diverse sketch styles. Our key observation is that the style descriptions of a sketch in natural language preserve the style during automatic sketch completion. Thus, we use a pretrained vision-language model (VLM) to describe the styles of the partial sketches in natural language and replicate these styles using newly generated strokes. We initially optimize the strokes to match an input prompt augmented by style descriptions extracted from the VLM. Such descriptions allow the method to establish a diffusion prior in close alignment with that of the partial sketch. Next, we utilize the VLM to generate an executable style adjustment code that adjusts the strokes to conform to the desired style. We compare our method with existing methods across various sketch styles and prompts, performed extensive ablation studies and qualitative and quantitative evaluations, and demonstrate that AutoSketch can support various sketch scenarios.

Problem

• AutoSketch completes partial sketches
• Preserves original sketch styles
• Uses VLM for style replication

Innovation

• VLM-assisted sketch completion
• Style-aware vector strokes
• Diffusion prior alignment