Envisioning Future Interactive Web Development: Editing Webpage with Natural Language

2025-10-30

AI Summary

Current web code editing relies on manual iteration, and large language models (LLMs) struggle with natural-language-driven precise editing due to the scarcity of high-quality, fine-tuning data. Method: We propose an automated data synthesis pipeline to construct Instruct4Edit—the first large-scale, visually verified web editing instruction dataset. It leverages LLMs to generate diverse editing instructions, applies structure-aware code modifications, and enforces end-to-end visual validation to ensure semantic fidelity and rendering accuracy. Contribution/Results: Fine-tuning open-source models on Instruct4Edit achieves performance on par with commercial systems for natural-language-driven web editing, significantly outperforming existing baselines. To foster reproducibility and further research, we fully open-source the dataset, training code, and fine-tuned models.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

The evolution of web applications relies on iterative code modifications, a process that is traditionally manual and time-consuming. While Large Language Models (LLMs) can generate UI code, their ability to edit existing code from new design requirements (e.g., "center the logo") remains a challenge. This is largely due to the absence of large-scale, high-quality tuning data to align model performance with human expectations. In this paper, we introduce a novel, automated data generation pipeline that uses LLMs to synthesize a high-quality fine-tuning dataset for web editing, named Instruct4Edit. Our approach generates diverse instructions, applies the corresponding code modifications, and performs visual verification to ensure correctness. By fine-tuning models on Instruct4Edit, we demonstrate consistent improvement in translating human intent into precise, structurally coherent, and visually accurate code changes. This work provides a scalable and transparent foundation for natural language based web editing, demonstrating that fine-tuning smaller open-source models can achieve competitive performance with proprietary systems. We release all data, code implementations, and model checkpoints for reproduction.

Problem

• Editing existing webpage code using natural language instructions remains challenging
• Lack of high-quality training data hinders LLMs' web editing capabilities
• Need to translate human design requirements into precise code modifications

Innovation

• Automated pipeline generates high-quality web editing dataset
• LLM-synthesized data enables precise code modification alignment
• Fine-tuned open models achieve competitive visual editing performance