SVGenius: Benchmarking LLMs in SVG Understanding, Editing and Generation

2025-06-03

AI Summary

Existing SVG benchmarks suffer from narrow real-world coverage, lack of complexity stratification, and fragmented evaluation dimensions, hindering systematic assessment of LLMs/MMLMs on SVG understanding, editing, and generation. Method: We introduce SVGenius—the first comprehensive SVG-processing benchmark—comprising 2,377 real-world queries, eight task categories, 18 fine-grained metrics, coverage across 24 application domains, and explicit complexity stratification. We propose a novel unified “understanding-editing-generation” tri-dimensional evaluation framework. Contribution/Results: SVGenius reveals a pronounced performance degradation in models with increasing SVG complexity; demonstrates that reasoning-augmented training outperforms mere parameter scaling; and identifies style transfer as the fundamental bottleneck. Cross-architectural evaluation of 22 state-of-the-art models shows: (i) closed-source models lead overall, yet all exhibit sharp performance drops on high-complexity tasks; (ii) reasoning-augmented variants yield significant gains; and (iii) style transfer accuracy remains the lowest, demanding urgent advancement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large Language Models (LLMs) and Multimodal LLMs have shown promising capabilities for SVG processing, yet existing benchmarks suffer from limited real-world coverage, lack of complexity stratification, and fragmented evaluation paradigms. We introduce SVGenius, a comprehensive benchmark comprising 2,377 queries across three progressive dimensions: understanding, editing, and generation. Built on real-world data from 24 application domains with systematic complexity stratification, SVGenius evaluates models through 8 task categories and 18 metrics. We assess 22 mainstream models spanning different scales, architectures, training paradigms, and accessibility levels. Our analysis reveals that while proprietary models significantly outperform open-source counterparts, all models exhibit systematic performance degradation with increasing complexity, indicating fundamental limitations in current approaches; however, reasoning-enhanced training proves more effective than pure scaling for overcoming these limitations, though style transfer remains the most challenging capability across all model types. SVGenius establishes the first systematic evaluation framework for SVG processing, providing crucial insights for developing more capable vector graphics models and advancing automated graphic design applications. Appendix and supplementary materials (including all data and code) are available at https://zju-real.github.io/SVGenius.

Problem

• Assessing LLMs' SVG understanding, editing, and generation capabilities
• Addressing limited real-world coverage and complexity stratification in benchmarks
• Evaluating model performance degradation with increasing task complexity

Innovation

• Comprehensive benchmark with 2,377 queries
• Evaluates models through 8 task categories
• Systematic complexity stratification for assessment