Affordances of Sketched Notations for Multimodal UI Design and Development Tools

2025-08-12

AI Summary

Existing sketch recognition methods struggle to interpret unconstrained, context-dependent hand-drawn UI sketches, hindering usability and intuitiveness in multimodal UI design tools. Method: We propose the novel principle “training data as symbol specification,” analyzing symbol properties of constrained versus freehand sketches through the Cognitive Dimensions Framework; we further integrate Transformer architectures with human-in-the-loop reinforcement learning to build a context-aware AI parsing model. Contribution/Results: We find that freehand sketches—though superficially ambiguous—are contextually resolvable and impose lower cognitive load than constrained alternatives. Critically, conventional element-level recognition paradigms exhibit fundamental limitations for creative sketching; in contrast, our context-sensitive AI approach significantly improves parsing accuracy for expressive sketches and enhances tool usability and human-centeredness.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multimodal UI design and development tools that interpret sketches or natural language descriptions of UIs inherently have notations: the inputs they can understand. In AI-based systems, notations are implicitly defined by the data used to train these systems. In order to create usable and intuitive notations for interactive design systems, we must regard, design, and evaluate these training datasets as notation specifications. To better understand the design space of notational possibilities for future design tools, we use the Cognitive Dimensions of Notations framework to analyze two possible notations for UI sketching. The first notation is the sketching rules for an existing UI sketch dataset, and the second notation is the set of sketches generated by participants in this study, where individuals sketched UIs without imposed representational rules. We imagine two systems, FixedSketch and FlexiSketch, built with each notation respectively, in order to understand the differential affordances of, and potential design requirements for, systems. We find that participants' sketches were composed of element-level notations that are ambiguous in isolation but are interpretable in context within whole designs. For many cognitive dimensions, the FlexiSketch notation supports greater intuitive creative expression and affords lower cognitive effort than the FixedSketch notation, but cannot be supported with prevailing, element-based approaches to UI sketch recognition. We argue that for future multimodal design tools to be truly human-centered, they must adopt contemporary AI methods, including transformer-based and human-in-the-loop, reinforcement learning techniques to understand users' context-rich expressive notations and corrections.

Problem

• Analyzing notations for multimodal UI design tools
• Comparing FixedSketch and FlexiSketch notations
• Exploring AI methods for intuitive UI sketch recognition

Innovation

• Uses Cognitive Dimensions of Notations framework
• Compares FixedSketch and FlexiSketch notations
• Adopts transformer-based and reinforcement learning