Video2Code: Generating Interactive Webpages from UI Videos via Action-Aware Revisit

2026-06-16

AI Summary

Existing approaches for translating UI videos into code struggle to accurately capture action boundaries due to sparse temporal representations, leading to misaligned state-action-state transitions and failure to reconstruct authentic user interactions. This work proposes Video2Code, which formulates code generation as an executable state-transition recovery task. It introduces a novel action-aware temporal revisiting mechanism: first identifying key interaction regions at a coarse granularity, then revisiting them with high temporal resolution for fine-grained refinement. By integrating action-aligned video-to-code supervised training with fine-grained temporal cropping, the method substantially enhances the temporal modeling capacity of vision-language models. Experiments demonstrate that Video2Code significantly outperforms existing methods in functional correctness, particularly in complex, multi-step interactive scenarios.

Abstract

UI videos provide a natural input for generating interactive webpages, as they capture both webpage appearance and action-triggered state transitions. However, directly applying video-capable vision-language models to this task remains insufficient. Existing models typically rely on sparse sampling or compressed temporal representations, which may miss short action boundaries and break the state-action-state transitions needed to implement webpage behavior. We formulate UI video-to-code generation as executable state-transition recovery from interaction videos, and identify this failure mode as state-transition misalignment. We introduce Video2Code, an action-aware video-to-code approach for recovering executable UI state transitions. Rather than allocating the visual budget uniformly across the video, Video2Code first performs coarse video understanding to locate action-critical regions, then invokes a temporal clipping tool to revisit these regions at higher temporal resolution before generating HTML/CSS/JavaScript code. We instantiate Video2Code with action-aligned video-code supervision and evaluate it under both visual and functional criteria. Experiments show that Video2Code substantially strengthens the underlying open-source model for UI video-to-code generation, improving functional correctness over direct video observation, especially on dense multi-step interactions.

Problem

• UI video-to-code
• state-transition misalignment
• interactive webpages
• action-aware
• executable behavior

Innovation

• action-aware
• state-transition recovery
• temporal clipping
• UI video-to-code
• functional correctness