Controlling Your Image via Simplified Vector Graphics

2026-02-16

AI Summary

Existing image generation methods struggle to achieve fine-grained control over shape, color, and semantics at the element level. This work proposes a hierarchical, controllable image generation framework based on simplified vector graphics (VG). The approach first parses an image into a semantically aligned and structurally coherent hierarchical VG representation, then employs a VG-guided noise prediction mechanism to seamlessly map user edits on vector elements to high-fidelity images. By leveraging simplified VG as a novel guidance signal for photorealistic image synthesis, this method enables unprecedented fine-grained controllability across geometric, chromatic, and semantic dimensions, establishing a new paradigm for editable generative modeling. Experiments demonstrate that the framework offers intuitive, efficient, and high-quality performance in object-level manipulation, local editing, and content creation tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in image generation have achieved remarkable visual quality, while a fundamental challenge remains: Can image generation be controlled at the element level, enabling intuitive modifications such as adjusting shapes, altering colors, or adding and removing objects? In this work, we address this challenge by introducing layer-wise controllable generation through simplified vector graphics (VGs). Our approach first efficiently parses images into hierarchical VG representations that are semantic-aligned and structurally coherent. Building on this representation, we design a novel image synthesis framework guided by VGs, allowing users to freely modify elements and seamlessly translate these edits into photorealistic outputs. By leveraging the structural and semantic features of VGs in conjunction with noise prediction, our method provides precise control over geometry, color, and object semantics. Extensive experiments demonstrate the effectiveness of our approach in diverse applications, including image editing, object-level manipulation, and fine-grained content creation, establishing a new paradigm for controllable image generation. Project page: https://guolanqing.github.io/Vec2Pix/

Problem

• controllable image generation
• element-level control
• image editing
• vector graphics
• object manipulation

Innovation

• vector graphics
• controllable generation
• semantic-aligned representation
• image editing
• hierarchical synthesis