SketchRef: a Multi-Task Evaluation Benchmark for Sketch Synthesis

2024-08-16

AI Summary

Existing sketch synthesis research lacks a unified evaluation benchmark, leading to unfair and incomplete method comparisons. Method: We introduce SketchRef, the first multi-task benchmark for sketch synthesis, covering four domains—animals, everyday objects, human bodies, and faces—and defining two core tasks: category prediction and structural consistency estimation (comprising five subtasks). We propose mRS, a novel quantitative metric that jointly captures recognizability and simplicity, revealing their intrinsic trade-off—the first such structural consistency estimation task in sketch synthesis. Evaluation leverages shared visual representations between sketches and reference photographs, integrating 7,920 human annotations with computational metrics for multidimensional validation. Contribution/Results: Our systematic evaluation of mainstream methods delineates their capabilities and limitations; we publicly release SketchRef to standardize evaluation and accelerate algorithmic progress.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Sketching is a powerful artistic technique for capturing essential visual information about real-world objects and has increasingly attracted attention in image synthesis research. However, the field lacks a unified benchmark to evaluate the performance of various synthesis methods. To address this, we propose SketchRef, the first comprehensive multi-task evaluation benchmark for sketch synthesis. SketchRef fully leverages the shared characteristics between sketches and reference photos. It introduces two primary tasks: category prediction and structural consistency estimation, the latter being largely overlooked in previous studies. These tasks are further divided into five sub-tasks across four domains: animals, common things, human body, and faces. Recognizing the inherent trade-off between recognizability and simplicity in sketches, we are the first to quantify this balance by introducing a recognizability calculation method constrained by simplicity, mRS, ensuring fair and meaningful evaluations. To validate our approach, we collected 7,920 responses from art enthusiasts, confirming the effectiveness of our proposed evaluation metrics. Additionally, we evaluate the performance of existing sketch synthesis methods on our benchmark, highlighting their strengths and weaknesses. We hope this study establishes a standardized benchmark and offers valuable insights for advancing sketch synthesis algorithms.

Problem

• Lacks unified benchmark for sketch synthesis evaluation
• Proposes SketchRef for multi-task sketch synthesis assessment
• Quantifies recognizability-simplicity trade-off in sketch evaluation

Innovation

• Introduces SketchRef benchmark for sketch synthesis
• Quantifies recognizability balance with mRS method
• Evaluates synthesis methods across four domains