UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization

2026-06-30

AI Summary

Existing vision-to-code generation methods struggle to achieve pixel-level precise alignment due to insufficient supervised fine-tuning and coarse rewards with stagnant exploration in reinforcement learning. This work proposes a unified reinforcement learning framework that introduces a novel symbolic attribute alignment mechanism to provide element-level dense rewards and designs a reference-guided code refinement strategy to transform blind exploration into guided policy improvement. By integrating a lightweight auxiliary LLM for parsing code attributes and injecting dynamic ground-truth execution trajectories, the approach significantly enhances generation accuracy and generalization. The method surpasses all existing open-source models across multiple benchmarks, attaining state-of-the-art performance on par with closed-source counterparts and advancing a new paradigm for general-purpose vision-to-code synthesis.

Abstract

Visual-to-Code generation, which transforms scientific plots, vector graphics, and webpages into executable scripts, demands a level of pixel-precise alignment that standard Multimodal Large Language Models (MLLMs) fail to achieve through Supervised Fine-Tuning (SFT) alone. While Reinforcement Learning (RL) offers a theoretical pathway to bridge this gap, its application is hindered by two fundamental obstacles: (1) \textit{Reward Coarseness}, where semantic metrics like CLIP scores fail to penalize fine-grained element deviations, and (2) \textit{Exploration Stagnation}, where the sparse, heterogeneous code search space prevents the policy from bootstrapping valid trajectories. To overcome these limitations, we introduce UniCoder, a unified RL framework that integrates two novel mechanisms. First, we propose \textbf{Symbolic Attribute Alignment}, which employs a lightweight auxiliary LLM to parse generated code into discrete visual attributes (e.g., hex colors, coordinate limits), enabling dense, element-wise reward computation. Second, to escape local optima, we devise \textbf{Reference-Guided Code Optimization}, a strategy that dynamically injects ground-truth trajectories into low-performing rollout groups, transforming blind exploration into guided policy improvement. Extensive experiments on ChartMimic, UniSVG, Design2Code and ScreenBench benchmarks demonstrate that our 8B-parameter model not only surpasses all open-source baselines but also achieves state-of-the-art performance comparable to proprietary models, establishing a new paradigm for generalized visual-to-code synthesis.

Problem

• Visual-to-Code Generation
• Reward Coarseness
• Exploration Stagnation
• Pixel-Precise Alignment
• Multimodal Large Language Models

Innovation

• Symbolic Attribute Alignment
• Reference-Guided Code Optimization
• Visual-to-Code Generation
• Reinforcement Learning
• Multimodal LLMs