Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback

2026-04-22

AI Summary

Existing SVG generation methods rely on a “blind drawing” strategy that lacks awareness of intermediate visual states, making it difficult to handle visually explicit yet textually ambiguous challenges such as partial visibility and occlusion. This work proposes a Render-in-the-Loop paradigm that formulates SVG generation as an iterative, visually context-aware process. By rendering the cumulative canvas at each step, the model can observe the evolving visual state and dynamically adjust its output. The approach incorporates Visual Self-Feedback (VSF) training and a Render-and-Verify (RaV) inference mechanism to enable real-time visual feedback–driven control. Built upon a multimodal large language model and fine-grained path decomposition, the method significantly outperforms strong baselines on MMSVGBench, demonstrating superior data efficiency and generalization in both Text-to-SVG and Image-to-SVG tasks.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) have shown promising capabilities in generating Scalable Vector Graphics (SVG) via direct code synthesis. However, existing paradigms typically adopt an open-loop "blind drawing" approach, where models generate symbolic code sequences without perceiving intermediate visual outcomes. This methodology severely underutilizes the powerful visual priors embedded in MLLMs vision encoders, treating SVG generation as a disjointed textual sequence modeling task rather than an integrated visuo-spatial one. Consequently, models struggle to reason about partial canvas states and implicit occlusion relationships, which are visually explicit but textually ambiguous. To bridge this gap, we propose Render-in-the-Loop, a novel generation paradigm that reformulates SVG synthesis as a step-wise, visual-context-aware process. By rendering intermediate code states into a cumulative canvas, the model explicitly observes the evolving visual context at each step, leveraging on-the-fly feedback to guide subsequent generation. However, we demonstrate that applying this visual loop naively to off-the-shelf models is suboptimal due to their inability to leverage incremental visual-code mappings. To address this, we first utilize fine-grained path decomposition to construct dense multi-step visual trajectories, and then introduce a Visual Self-Feedback (VSF) training strategy to condition the next primitive generation on intermediate visual states. Furthermore, a Render-and-Verify (RaV) inference mechanism is proposed to effectively filter degenerate and redundant primitives. Our framework, instantiated on a multimodal foundation model, outperforms strong open-weight baselines on the standard MMSVGBench. This result highlights the remarkable data efficiency and generalization capability of our Render-in-the-Loop paradigm for both Text-to-SVG and Image-to-SVG tasks.

Problem

• SVG generation
• visual feedback
• multimodal large language models
• vector graphics
• visuo-spatial reasoning

Innovation

• Render-in-the-Loop
• Visual Self-Feedback
• SVG Generation
• Multimodal Large Language Models
• Render-and-Verify