From Visual Widgets to UI Code: Efficient Tool-Grounded Generation

2026-08-12

AI Summary

Existing screenshot-to-code approaches struggle to balance flexibility and controllability: end-to-end generation often suffers from hallucination, while structured pipelines constrain design expressiveness. This work proposes WidgetGen, a lightweight, tool-anchored framework that selectively invokes modular components—such as text/color extraction, layout reasoning, and chart recognition—to directly synthesize executable JSX code. By avoiding rigid UI templates and reducing reliance on component-level generation, WidgetGen achieves greater adaptability. Experimental results demonstrate that WidgetGen significantly outperforms both direct prompting and the Widget2Code pipeline across 1,000 held-out UI controls, yielding consistent improvements in region accuracy, code readability, and style fidelity. Furthermore, supervised fine-tuning with WidgetGen effectively enhances the code generation capabilities of Qwen-family models.

Abstract

Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce \textbf{WidgetGen}, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (\emph{JSX}). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and \(1{,}000\) held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.

Problem

• screenshot-to-code
• UI generation
• multimodal generation
• code fidelity
• widget-to-code

Innovation

• tool-grounded generation
• screenshot-to-code
• JSX generation
• visual fidelity
• multimodal UI understanding