Towards Human-AI Synergy in UI Design: Enhancing Multi-Agent Based UI Generation with Intent Clarification and Alignment

2024-12-28

AI Summary

Current UI auto-design systems lack iterative refinement capabilities, primarily due to inaccurate design intent interpretation and opaque generation processes. To address this, we propose PrototypeAgent, a multi-agent framework enabling designers to drive prototype generation via natural language instructions and layout preferences. Its novel “intent clarification–dynamic alignment” mechanism enables real-time inference of implicit requirements and interactive correction of intermediate artifacts during generation, facilitating human-AI collaborative iteration. Technically, the framework integrates LLM prompt enhancement, a topic-guided agent coordination architecture, editable intermediate-result interfaces, and a multi-stage intent alignment evaluation module. Extensive experiments and user studies demonstrate that PrototypeAgent significantly outperforms state-of-the-art baselines in prototype fidelity, design accuracy, and output diversity.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Multiagent Systems: Mechanism Design
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

In automated user interface (UI) design generation, a key challenge is the lack of support for iterative processes, as most systems only focus on end-to-end generation of designs as starting points. This results from (1) limited capabilities to fully interpret user design intent from text or images, and (2) a lack of transparency, which prevents designers from refining intermediate results. To address existing limitations, we introduce PrototypeAgent, a human-centered, multi-agent system for automated UI generation. The core of PrototypeAgent is a theme design agent that clarifies implicit design intent through prompt augmentation, coordinating with specialized sub-agents to generate specific components. Designers interact with the system via an intuitive interface, providing natural language descriptions and layout preferences. During generation, PrototypeAgent enables designers to refine generated intermediate guidance or specific components, ensuring alignment with their intent throughout the generation workflow. Evaluations through experiments and user studies show PrototypeAgent's effectiveness in producing high-fidelity prototypes that accurately reflect design intent as well as its superiority over baseline models in terms of both quality and diversity.

Problem

• Automated UI Design
• Iterative Revision
• User Preferences

Innovation

• PrototypeAgent
• Human-Computer Collaboration
• Enhanced Prompt Understanding