Hey Dashboard!: Supporting Voice, Text, and Pointing Modalities in Dashboard Onboarding

2025-10-14

AI Summary

Contemporary dashboards suffer from complex interactions and tightly coupled views, necessitating labor-intensive authoring and maintenance of guided tutorials—resulting in high costs and poor synchronization with dashboard updates. To address this, we propose DIANA, the first multimodal dashboard assistant integrating speech, text, and mouse gaze inputs. Built upon large language models (LLMs), DIANA implements a context-aware, real-time interactive system that enables users to issue queries via any combination of modalities and receive immediate visual feedback—including interface element highlighting—and semantically grounded explanations. Its key innovation lies in transcending the conventional unimodal text-based LLM paradigm by enabling synergistic, tri-modal-driven dynamic guidance. DIANA establishes, for the first time in visualization analytics, a closed-loop pipeline spanning multimodal input, interface response, and semantic output. A user study demonstrates that DIANA significantly improves users’ comprehension efficiency of dashboard structure and functionality while markedly reducing reliance on manual guidance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Visualization dashboards are regularly used for data exploration and analysis, but their complex interactions and interlinked views often require time-consuming onboarding sessions from dashboard authors. Preparing these onboarding materials is labor-intensive and requires manual updates when dashboards change. Recent advances in multimodal interaction powered by large language models (LLMs) provide ways to support self-guided onboarding. We present DIANA (Dashboard Interactive Assistant for Navigation and Analysis), a multimodal dashboard assistant that helps users for navigation and guided analysis through chat, audio, and mouse-based interactions. Users can choose any interaction modality or a combination of them to onboard themselves on the dashboard. Each modality highlights relevant dashboard features to support user orientation. Unlike typical LLM systems that rely solely on text-based chat, DIANA combines multiple modalities to provide explanations directly in the dashboard interface. We conducted a qualitative user study to understand the use of different modalities for different types of onboarding tasks and their complexities.

Problem

• Reducing labor-intensive dashboard onboarding preparation
• Supporting self-guided multimodal interaction for dashboard exploration
• Integrating voice text and pointing for dashboard orientation

Innovation

• Multimodal assistant combining voice, text, pointing
• LLM-powered interactive guidance for dashboard navigation
• Cross-modality explanations directly in dashboard interface