Controllable GUI Exploration

2025-02-05

AI Summary

Early GUI design requires rapid generation of diverse low-fidelity sketches to explore a broad design space; however, existing tools demand overly precise inputs, while pure text prompts struggle to encode loosely specified design intents. This paper proposes a diffusion-based multimodal controllable sketch generation method. It introduces a novel progressive conditional control paradigm that flexibly integrates three optional input modalities—textual descriptions, wireframes, and interactive flowcharts—to enable fine-grained, low-overhead synthesis of低保真 GUI sketches. Experiments demonstrate that our method significantly improves constraint alignment accuracy and generation diversity across diverse input combinations. Compared to baseline models, it enables more efficient large-scale design exploration. To the best of our knowledge, this is the first generative framework for early-stage interface design that jointly achieves flexibility, precise controllability, and practical usability.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic aspects and information design of Web, GenAI, and cloud computing
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

During the early stages of interface design, designers need to produce multiple sketches to explore a design space. Design tools often fail to support this critical stage, because they insist on specifying more details than necessary. Although recent advances in generative AI have raised hopes of solving this issue, in practice they fail because expressing loose ideas in a prompt is impractical. In this paper, we propose a diffusion-based approach to the low-effort generation of interface sketches. It breaks new ground by allowing flexible control of the generation process via three types of inputs: A) prompts, B) wireframes, and C) visual flows. The designer can provide any combination of these as input at any level of detail, and will get a diverse gallery of low-fidelity solutions in response. The unique benefit is that large design spaces can be explored rapidly with very little effort in input-specification. We present qualitative results for various combinations of input specifications. Additionally, we demonstrate that our model aligns more accurately with these specifications than other models.

Problem

• low-effort interface sketch generation
• flexible control via multiple inputs
• rapid exploration of design spaces

Innovation

• Diffusion-based sketch generation
• Flexible control via inputs
• Rapid low-effort design exploration