AgentBuilder: Exploring Scaffolds for Prototyping User Experiences of Interface Agents

2025-10-05

AI Summary

Non-AI engineers lack efficient tools to prototype generative AI-based UI Agent experiences. Method: We propose AgentBuilder—a user-centered design framework and lightweight prototyping tool—developed through contextual inquiry, design probes, and in-situ experiments to identify core activities and capability requirements for agent experience design. AgentBuilder supports low-code interaction orchestration, real-time LLM integration, and iterative multi-turn dialogue prototyping. Evaluation with 14 cross-disciplinary participants demonstrated that the framework significantly lowers prototyping barriers, enhances designer engagement, and improves feedback quality. It further revealed non-technical users’ critical needs for controllability, explainability, and progressive guidance. Contribution/Results: This work presents the first systematic methodology and practical toolchain for UI Agent experience prototyping tailored specifically for non-engineers, bridging a critical gap between human-centered design and generative AI interface development.

Technology Category

• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent/AI Theories and Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Interface agents powered by generative AI models (referred to as "agents") can automate actions based on user commands. An important aspect of developing agents is their user experience (i.e., agent experience). There is a growing need to provide scaffolds for a broader set of individuals beyond AI engineers to prototype agent experiences, since they can contribute valuable perspectives to designing agent experiences. In this work, we explore the affordances agent prototyping systems should offer by conducting a requirements elicitation study with 12 participants with varying experience with agents. We identify key activities in agent experience prototyping and the desired capabilities of agent prototyping systems. We instantiate those capabilities in the AgentBuilder design probe for agent prototyping. We conduct an in situ agent prototyping study with 14 participants using AgentBuilder to validate the design requirements and elicit insights on how developers prototype agents and what their needs are in this process.

Problem

• Exploring scaffolds for prototyping interface agent experiences
• Identifying key activities in agent experience prototyping process
• Developing AgentBuilder probe to validate design requirements

Innovation

• AgentBuilder enables agent prototyping for diverse users
• System scaffolds agent experience design process
• Design probe validates prototyping requirements and capabilities