Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation

2025-10-10

AI Summary

Natural language instructions for data visualization code generation often suffer from semantic ambiguity, leading to generated code that deviates from users’ true intent. Method: We propose a visualization-oriented ambiguity taxonomy and computable quantitative metrics, grounded in formal pragmatics—integrating Grice’s Cooperative Principle, Discourse Representation Theory, and the Question Under Discussion (QUD) model—to design a pragmatic-driven, multi-turn clarification dialogue mechanism. Contribution/Results: Evaluated on Matplotlib code generation using the DS-1000 benchmark, our ambiguity metrics achieve strong inter-annotator agreement with human judgments (Cohen’s κ = 0.82), significantly outperforming baselines. A user simulation study shows that our interactive refinement approach improves code correctness by 23.7% and achieves an ambiguity resolution rate of 89.4%. This work is the first to systematically incorporate formal pragmatics into human–AI collaboration for visualization code generation, establishing a novel paradigm for modeling ambiguous intent and enabling interactive, intention-aware code synthesis.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data visualization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Establishing shared goals is a fundamental step in human-AI communication. However, ambiguities can lead to outputs that seem correct but fail to reflect the speaker's intent. In this paper, we explore this issue with a focus on the data visualization domain, where ambiguities in natural language impact the generation of code that visualizes data. The availability of multiple views on the contextual (e.g., the intended plot and the code rendering the plot) allows for a unique and comprehensive analysis of diverse ambiguity types. We develop a taxonomy of types of ambiguity that arise in this task and propose metrics to quantify them. Using Matplotlib problems from the DS-1000 dataset, we demonstrate that our ambiguity metrics better correlate with human annotations than uncertainty baselines. Our work also explores how multi-turn dialogue can reduce ambiguity, therefore, improve code accuracy by better matching user goals. We evaluate three pragmatic models to inform our dialogue strategies: Gricean Cooperativity, Discourse Representation Theory, and Questions under Discussion. A simulated user study reveals how pragmatic dialogues reduce ambiguity and enhance code accuracy, highlighting the value of multi-turn exchanges in code generation.

Problem

• Identifying ambiguous user goals in visualization code generation
• Developing taxonomy and metrics to quantify visualization ambiguities
• Exploring dialogue strategies to reduce ambiguity in code generation

Innovation

• Developed ambiguity taxonomy and quantification metrics
• Utilized multi-turn dialogue to refine user goals
• Evaluated pragmatic models for dialogue strategies