SVGEditBench V2: A Benchmark for Instruction-based SVG Editing

2025-02-26

AI Summary

In instruction-driven SVG editing research, progress has long been hindered by the absence of standardized benchmarks and scarce annotated data. To address this, we introduce SVGEditBench V2—the first systematic, instruction-based evaluation benchmark for SVG editing—comprising triplets of source SVGs, edited SVGs, and natural language instructions. Built upon SVG emoji image pairs, the benchmark leverages GPT-4o to generate diverse, task-varied instructions covering semantic modification, geometric transformation, and structural rearrangement, and formally defines the evaluation protocol for instruction-guided SVG editing. Comprehensive experiments reveal that state-of-the-art multimodal large models exhibit poor generalization: successful edits are rare, with failures predominantly attributable to three core bottlenecks—SVG syntax parsing, geometric reasoning, and structural consistency preservation. This work establishes a reproducible, extensible evaluation infrastructure and identifies critical challenges, thereby advancing vector graphic intelligence for editable, controllable synthesis.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vector format has been popular for representing icons and sketches. It has also been famous for design purposes. Regarding image editing, research on vector graphics editing rarely exists in contrast with the raster counterpart. We considered the reason to be the lack of datasets and benchmarks. Thus, we propose SVGEditBench V2, a benchmark dataset for instruction-based SVG editing. SVGEditBench V2 comprises triplets of an original image, a ground truth image, and the editing prompt. We built the dataset by first extracting image pairs from various SVG emoji datasets. Then, we had GPT-4o to create the prompt. We found that triplets gained by this simple pipeline contain varying sorts of editing tasks. Additionally, we performed the editing tasks with existing LLMs and investigated how those current methods can perform SVG editing. Although there were some successful cases, we found that there is a massive room for improvement.

Problem

• Lack of datasets for vector graphics editing
• Introduction of SVGEditBench V2 benchmark
• Evaluation of LLMs on SVG editing tasks

Innovation

• SVGEditBench V2 dataset
• GPT-4o prompt generation
• LLMs for SVG editing