Misty: UI Prototyping Through Interactive Conceptual Blending

2024-09-20

arXiv.org

AI Summary

Existing UI prototyping tools provide weak support for integrating design artifacts such as screenshots and sketches, hindering component reuse, semantic integration, and cross-role collaboration. This paper proposes a novel UI prototyping paradigm grounded in Conceptual Blending Theory, the first to concretize cognitive-science-based blending mechanisms into an interactive tool. It enables semantic-level element mixing across heterogeneous design examples through example-driven component extraction and semantic alignment, lightweight vision–semantics mapping, and real-time blended preview—facilitating staged intent articulation by developers. An empirical study with 14 frontend developers demonstrates that the approach significantly reduces prototype initiation time (average improvement of 42%), stimulates highly unexpected creative combinations (68% novel composition rate), and enhances design–development collaboration efficiency.

Technology Category

• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Segmentation

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporate diverse aspects from design examples into work-in-progress UIs. We prototyped this workflow as Misty. Through an exploratory first-use study with 14 frontend developers, we assessed Misty's effectiveness and gathered feedback on this workflow. Our findings suggest that Misty's conceptual blending workflow helps developers kickstart creative explorations, flexibly specify intent in different stages of prototyping, and inspires developers through serendipitous UI blends. Misty demonstrates the potential for tools that blur the boundaries between developers and designers.

Problem

• Limited tool support for blending design examples in UI prototyping
• Need for flexible intent specification across prototyping stages
• Bridging gaps between developer and designer workflows

Innovation

• UI workflow blending diverse design examples
• Misty supports flexible intent specification
• Inspires creativity through serendipitous UI blends