Bridging Gulfs in UI Generation through Semantic Guidance

2026-01-27

AI Summary

This work addresses the execution–evaluation gap in generative AI–assisted UI design, where users often struggle to precisely articulate design intent and assess generated outputs. To bridge this gap, the paper proposes a novel interaction paradigm centered on explicit, structured semantic representations. By analyzing UI generation prompt guidelines, the authors extract a hierarchical and interdependent set of design semantics and implement a system that enables users to explicitly specify these semantics, visualize their relationships, and trace their manifestation in generated results. This approach introduces structured semantic representation into AI-driven UI generation for the first time, rendering design intent expressible, outputs interpretable, and iterative refinement predictable. User studies demonstrate that the system significantly enhances users’ perceived capabilities in expressing intent, understanding results, and controlling iterative design, thereby enabling more efficient and explainable AI-assisted UI design.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Natural Language Processing: Generation
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While generative AI enables high-fidelity UI generation from text prompts, users struggle to articulate design intent and evaluate or refine results-creating gulfs of execution and evaluation. To understand the information needed for UI generation, we conducted a thematic analysis of UI prompting guidelines, identifying key design semantics and discovering that they are hierarchical and interdependent. Leveraging these findings, we developed a system that enables users to specify semantics, visualize relationships, and extract how semantics are reflected in generated UIs. By making semantics serve as an intermediate representation between human intent and AI output, our system bridges both gulfs by making requirements explicit and outcomes interpretable. A comparative user study suggests that our approach enhances users'perceived control over intent expression and outcome interpretation, and facilitates more predictable iterative refinement. Our work demonstrates how explicit semantic representation enables systematic and explainable exploration of design possibilities in AI-driven UI design.

Problem

• UI generation
• generative AI
• design intent
• gulf of execution
• gulf of evaluation

Innovation

• semantic guidance
• UI generation
• generative AI
• design semantics
• human-AI interaction