A Taxonomy of Human--MLLM Interaction in Early-Stage Sketch-Based Design Ideation

2026-02-25

AI Summary

This study investigates how designers collaborate with multimodal large language models (MLLMs) during the early sketching phase of ideation and how creative agency is dynamically allocated between human and AI. Drawing on interaction log analysis and semi-structured interviews, the research empirically examines the real-world usage of four predefined interaction modes: human-only, human-led, AI-led, and co-evolutionary. Findings reveal that designers rarely adhere to a single mode; instead, they frequently switch between human-led and AI-led approaches, highlighting the fluid and dynamic nature of human-AI collaboration. These insights challenge prevailing assumptions of static interaction paradigms and provide both theoretical grounding and empirical support for the development of future intelligent design tools that facilitate flexible role transitions between users and AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As multimodal large language models (MLLMs) are increasingly integrated into early-stage design tools, it is important to understand how designers collaborate with AI during ideation. In a user study with 12 participants, we analysed sketch-based design interactions with an MLLM-powered system using automatically recorded interaction logs and post-task interviews. Based on how creative responsibility was allocated between humans and the AI, we predefined four interaction modes: Human-Only, Human-Lead, AI-Lead, and Co-Evolution, and analysed how these modes manifested during sketch-based design ideation. Our results show that designers rarely rely on a single mode; instead, human-led and AI-led roles are frequently interwoven and shift across ideation instances. These findings provide an empirical basis for future work to investigate why designers shift roles with AI and how interactive systems can better support such dynamic collaboration.

Problem

• Human-AI collaboration
• design ideation
• sketch-based design
• creative responsibility
• interaction modes

Innovation

• Human-AI collaboration
• multimodal large language models
• design ideation
• interaction taxonomy
• sketch-based design