Qualitative Study for LLM-assisted Design Study Process: Strategies, Challenges, and Roles

2025-07-14

AI Summary

Despite growing adoption of large language models (LLMs) in visualization design research, there remains a lack of systematic empirical understanding of their practical roles and limitations. Method: We conducted a multi-stage qualitative study—including in-depth interviews and structured surveys—with 30 interdisciplinary researchers actively using LLMs in real-world visualization projects. Contribution/Results: We identify LLMs’ concrete functions, recurrent strategies, and shared challenges across key design phases—problem framing, data comprehension, and solution generation. Building on these insights, we propose “VizLLM,” the first comprehensive application framework that systematically integrates LLM-assisted mechanisms and evidence-informed practice principles throughout the end-to-end visualization design process. This work bridges a critical theoretical gap in LLM-augmented design research and delivers a reusable, empirically grounded methodology with actionable implementation pathways for researchers and practitioners.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Design studies aim to create visualization solutions for real-world problems of different application domains. Recently, the emergence of large language models (LLMs) has introduced new opportunities to enhance the design study process, providing capabilities such as creative problem-solving, data handling, and insightful analysis. However, despite their growing popularity, there remains a lack of systematic understanding of how LLMs can effectively assist researchers in visualization-specific design studies. In this paper, we conducted a multi-stage qualitative study to fill this gap, involving 30 design study researchers from diverse backgrounds and expertise levels. Through in-depth interviews and carefully-designed questionnaires, we investigated strategies for utilizing LLMs, the challenges encountered, and the practices used to overcome them. We further compiled and summarized the roles that LLMs can play across different stages of the design study process. Our findings highlight practical implications to inform visualization practitioners, and provide a framework for leveraging LLMs to enhance the design study process in visualization research.

Problem

• How LLMs enhance visualization design studies
• Strategies and challenges in LLM-assisted design
• Roles of LLMs in design study stages

Innovation

• Multi-stage qualitative study with diverse researchers
• In-depth interviews and designed questionnaires
• Framework for LLM roles in design stages