Capturing Visualization Design Rationale

2025-06-19

AI Summary

Existing natural language (NL) datasets for visualization primarily focus on decoding tasks—such as literacy assessment, insight extraction, or instruction generation—and rely on manually constructed charts and questions, neglecting explicit modeling of design intent. Method: This work introduces the first NL dataset targeting *encoding logic* understanding in visualization, built from authentic design decisions and their natural language rationales extracted from students’ visualization notebooks. It uniquely leverages pedagogical design explanations as supervision signals, integrating human annotation with LLM-driven generation, classification, and verification of question–answer–rationale triples. Contribution/Results: We release the first high-quality, interpretable dataset of visualization design rationales, enabling design intent modeling, visualization education assessment, and explainable-AI–driven visualization generation—thereby overcoming the explanatory limitations of prior NL-VIS datasets at the level of generative rationale.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Generation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data visualization

Abstract

Prior natural language datasets for data visualization have focused on tasks such as visualization literacy assessment, insight generation, and visualization generation from natural language instructions. These studies often rely on controlled setups with purpose-built visualizations and artificially constructed questions. As a result, they tend to prioritize the interpretation of visualizations, focusing on decoding visualizations rather than understanding their encoding. In this paper, we present a new dataset and methodology for probing visualization design rationale through natural language. We leverage a unique source of real-world visualizations and natural language narratives: literate visualization notebooks created by students as part of a data visualization course. These notebooks combine visual artifacts with design exposition, in which students make explicit the rationale behind their design decisions. We also use large language models (LLMs) to generate and categorize question-answer-rationale triples from the narratives and articulations in the notebooks. We then carefully validate the triples and curate a dataset that captures and distills the visualization design choices and corresponding rationales of the students.

Problem

• Capturing real-world visualization design rationale
• Moving beyond controlled setups to natural language narratives
• Using LLMs to generate and validate design rationale triples

Innovation

• Uses real-world literate visualization notebooks
• Leverages LLMs for question-answer-rationale generation
• Validates and curates design rationale dataset