Characterizing Multimodal Interaction in Visualization Authoring Tools

2025-09-10

AI Summary

The absence of a systematic characterization of multimodal interaction in contemporary visualization authoring tools hinders both design practice understanding and future research directions. Method: This study conducts the first cross-platform analysis of multimodal interaction features across 20 mainstream tools, grounded in human–computer interaction and information visualization theory. We systematically characterize input modality integration (e.g., gesture, speech, pen, eye tracking), coordination mechanisms, and design patterns. Contribution/Results: We identify common practices—such as modality complementarity to enhance expressive efficiency—and critical distinctions—e.g., professional tools favor structured input, while low-code platforms prioritize natural interaction. From this, we distill six canonical application patterns and three evolutionary trends. This work fills a critical gap in comprehensive multimodal analysis for visualization authoring and provides theoretical foundations and actionable design insights for developing more efficient, inclusive, and scalable multimodal visualization systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal interaction has been increasingly considered in designing visualization authoring tools. However, multimodal interaction has a broad meaning in visualization authoring, according to our literature review. Although some previous studies compare different authoring tools, a comprehensive overview of the diverse characteristics of multimodal interaction in visualization authoring tools is still missing. This paper seeks to offer a systematic perspective on how multimodal interaction is integrated within visualization authoring tools. Such an overview can enhance understanding of current practices, highlight distinguishing features among tools, and help identify future research directions, guiding designers in developing more accessible and effective authoring systems. We review 20 visualization authoring tools that incorporate multimodal interaction and characterize how multimodal interaction is applied in these tools. Based on the review results, we discuss design implications and future directions.

Problem

• Characterizing multimodal interaction in visualization authoring tools
• Providing systematic overview of multimodal integration practices
• Identifying design implications and future research directions

Innovation

• Systematic review of multimodal interaction tools
• Characterizes multimodal interaction in visualization authoring
• Identifies design implications and future directions