🤖 AI Summary
This study addresses the challenge of precisely repairing SVG code under visual instruction guidance, requiring models to modify only specified regions while preserving all other protected content. To this end, the authors introduce a benchmark comprising 40 high-difficulty tasks and propose a novel dual-norm reward mechanism that integrates semantic invariance with attribute-aware tolerance. They also introduce new evaluation dimensions, including validity-gated repair progress and Unintended Change Rate (UCR). Through rigorous assessment involving SVG parsing-rendering validation, structural-semantic consistency checks, and deterministic norm scoring across 34 models, they find that even the strongest model achieves a full-norm success rate of merely 15.0%, with an average repair progress of 43.7%, revealing significant limitations in current approaches to faithful, constrained editing.
📝 Abstract
Instruction-based vector editing requires two capabilities: making a requested change and leaving everything else alone. The second is easy to miss when an output is judged only as a raster image. We introduce Vector-Bench, a compact, difficult benchmark of 40 SVG repair tasks. Each task pairs a corrupted SVG program with an author-written visual instruction, a hidden target program, 5.05 annotated repairs on average, and an average of 60.55 protected objects. Instructions describe visible defects without exposing element identifiers, coordinates, color codes, or path data. We define a deterministic binary specification reward: requested repairs use attribute-aware perceptual tolerances, while unrequested rendering- or application-relevant structure must remain semantically unchanged and the result must be a valid SVG. Canonical target equality and stricter source fidelity are retained as diagnostics. Validity-gated repair progress, a near-complete tier, and valid-output Unintended Change Rate (UCR) explain partial outcomes. We evaluate 34 model endpoints (25 listed as open-weight, 5 inexpensive controls, and 4 frontier closed endpoints) over 1360 requests. The strongest endpoint reaches only 15.0% full specification success, despite 43.7% mean repair progress, showing that apparent repair progress and specification-faithful editing remain substantially different. All prompts, outputs, scoring code, costs, and per-task reports are released.