Talk Me Through It: Developing Effective Systems for Chart Authoring

2026-01-21

AI Summary

This study addresses a critical gap in existing chart generation systems, which predominantly rely on textual descriptions of pre-existing visualizations and thus struggle to support users’ intent to create charts from imagined designs conveyed through speech. The work presents the first systematic analysis of linguistic characteristics in voice-based charting instructions, revealing their distinctiveness in command structure, element specification, and linguistic complexity. Building upon these insights, the authors develop a chart generation system trained specifically on voice-imagined data. Through natural language processing, speech interaction protocols, and human-subject experiments, the system demonstrates significantly superior performance over conventional approaches under both spoken and typed inputs, underscoring the importance of aligning training data with the target interaction modality to achieve optimal performance.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

Recent chart-authoring systems increasingly focus on natural-language input, enabling users to form a mental image of the chart they wish to create and express this intent using spoken instructions (spoken imagined-chart data). Yet these systems are predominantly trained on typed instructions written while viewing the target chart (typed existing-chart data). While the cognitive processes for describing an existing chart arguably differ from those for creating a new chart, the structural differences in the corresponding prompts remain underexplored. We present empirical findings on the structural differences among spoken imagined-chart instructions, typed imagined-chart instructions, and typed existing-chart instructions for chart creation, showing that imagined-chart prompts contain richer command formats, element specifications, and complex linguistic features, especially in spoken instructions. We then compare the performance of systems trained on spoken imagined-chart data versus typed existing-chart data, finding that the first system outperforms the second one on both voice and text input, highlighting the necessity of targeted training on spoken imagined-chart data. We conclude with design guidelines for chart-authoring systems to improve performance in real-world scenarios.

Problem

• chart authoring
• natural language interface
• spoken instructions
• imagined-chart data
• existing-chart data

Innovation

• spoken imagined-chart data
• natural language chart authoring
• structural prompt differences
• targeted training
• voice-based visualization