wireframing

Designers often struggle to effectively apply insights from human-computer interaction (HCI) research due to challenges in retrieval, disciplinary terminology barriers, and a lack of contextualized, actionable guidance. To address this gap, this work proposes ReFinE—the first Figma plugin that automatically synthesizes HCI research findings and integrates them directly into the UI design workflow. Leveraging natural language processing and context-aware recommendation techniques, ReFinE delivers real-time, visual, and actionable design suggestions tailored to the designer’s current task. Findings from a user study demonstrate that ReFinE significantly reduces cognitive load, enhances designers’ ability to incorporate empirical research evidence into their practice, improves design quality, and accelerates iterative prototyping cycles.

Existing vision-language models (VLMs) lack systematic evaluation benchmarks for tool-driven UI design tasks—such as those performed in Figma or Sketch—hindering progress in professional design automation. Method: We introduce CANVAS, the first benchmark tailored for tool-augmented VLMs in UI design, comprising 598 context-aware design tasks across 30 mobile UI scenarios. It explicitly distinguishes “design reproduction” and “design modification” tasks, built upon real-world UI data with human-annotated ground truth. Our methodology employs a context-aware VLM–tool API co-execution framework for precise tool invocation. Contribution/Results: Experiments demonstrate that state-of-the-art VLMs exhibit nascent capability in collaborative tool-based design; however, strategic tool selection and design quality remain critical bottlenecks. CANVAS provides a reproducible, empirically grounded evaluation framework to advance VLM interaction, iterative refinement, and integration within professional design software.

Early GUI design requires rapid generation of diverse low-fidelity sketches to explore a broad design space; however, existing tools demand overly precise inputs, while pure text prompts struggle to encode loosely specified design intents. This paper proposes a diffusion-based multimodal controllable sketch generation method. It introduces a novel progressive conditional control paradigm that flexibly integrates three optional input modalities—textual descriptions, wireframes, and interactive flowcharts—to enable fine-grained, low-overhead synthesis of低保真 GUI sketches. Experiments demonstrate that our method significantly improves constraint alignment accuracy and generation diversity across diverse input combinations. Compared to baseline models, it enables more efficient large-scale design exploration. To the best of our knowledge, this is the first generative framework for early-stage interface design that jointly achieves flexibility, precise controllability, and practical usability.

Existing UI prototyping tools provide weak support for integrating design artifacts such as screenshots and sketches, hindering component reuse, semantic integration, and cross-role collaboration. This paper proposes a novel UI prototyping paradigm grounded in Conceptual Blending Theory, the first to concretize cognitive-science-based blending mechanisms into an interactive tool. It enables semantic-level element mixing across heterogeneous design examples through example-driven component extraction and semantic alignment, lightweight vision–semantics mapping, and real-time blended preview—facilitating staged intent articulation by developers. An empirical study with 14 frontend developers demonstrates that the approach significantly reduces prototype initiation time (average improvement of 42%), stimulates highly unexpected creative combinations (68% novel composition rate), and enhances design–development collaboration efficiency.

This work proposes an interactive UI design system that addresses the challenges non-expert users face in expressing design intent and trusting retrieved examples, which often lead to design fixation or disorientation. The system uniquely integrates multimodal retrieval-augmented generation (MMRAG) with source transparency, enabling fine-grained retrieval, remixing, and adaptation of UI examples at both interface and component levels. By transparently presenting metadata such as example ratings, download counts, and developer information, the system enhances users’ confidence in their design decisions. User studies demonstrate that this approach significantly improves users’ ability to achieve their design goals, facilitates efficient iteration, and encourages diverse exploration, effectively balancing controllability with creative discovery.

While large language models (LLMs) and coding agents are often applied to user interface (UI) development, developers find it difficult to reliably assess their proficiency in visual and interaction design. Existing evaluations either rely on human experts, who can accurately assess usability by testing critical flows but are slow and costly, or on automated judges, which are scalable but less accurate and opaque. We present FlowEval, a reference-based framework that measures whether a generated UI supports realistic interaction flows by comparing navigation traces from real websites to traces from generated analogs using reference-based similarity metrics (e.g., dynamic time warping). In a small-scale study with expert UI evaluators, we show that reference-based metrics strongly correlate with human judgments, suggesting that they can provide scalable yet trustworthy evaluation for UI generation systems.

This work addresses the creative stagnation often induced by existing generative design tools that directly output complete images. To overcome this limitation, the authors propose a multi-stage, compositional AI-assisted design approach that emulates professional designers’ workflows: it first structurally interprets ambiguous design requests, then generates candidate elements—such as objects, backgrounds, typography, layout, and composition—separately, and finally enables interactive recombination. This method formalizes real-world design processes into a computable system for the first time, decoupling requirement interpretation, element generation, and composition to substantially enhance prompt diversity and alignment with user intent. User studies demonstrate that the system outperforms baseline approaches in both requirement comprehension accuracy and designer-rated quality, revealing a productive trade-off between structured workflow, creative clarity, and efficiency despite slightly longer generation times.

This work addresses the inefficiency and lack of guidance faced by front-end developers when manually selecting plausible and natural attribute values for instantiating reusable UI components within a vast design space. To tackle this challenge, the paper introduces the concept of “discriminative variants,” which uniquely integrates symbolic reasoning with large language models (LLMs). Symbolic reasoning identifies visually salient attributes, while the LLM leverages real-world knowledge to generate component instances that balance fidelity to exemplars with meaningful differentiation. This approach shifts the paradigm from ad hoc manual configuration to structured exploration of the design space. A user study (n=12) demonstrates that the generated variants effectively aid developers in comprehending the design space, significantly improving both instantiation efficiency and user experience, while maintaining strong domain relevance.

This work proposes “annotation-driven animation,” a novel interaction paradigm that formalizes hand-drawn annotations—commonly used yet ambiguous, context-dependent, and unstructured in traditional animation—into a structured source-path-target representation. By integrating generative AI, the system automatically synthesizes keyframes from these annotations and employs dynamic UI controls to enable fine-grained user adjustment and disambiguation. The approach establishes a closed-loop human-AI collaboration framework that bridges imprecise sketches to executable animations. Preliminary user studies demonstrate its effectiveness and practicality in supporting efficient, intuitive keyframe generation and editing.