Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code

2024-12-03

arXiv.org

AI Summary

This work addresses the lack of standardized evaluation for large language models (LLMs) in generating visualization code from tabular data. We introduce PandasPlotBench, the first benchmark dataset specifically targeting Pandas DataFrame visualization—comprising 175 human-annotated tasks spanning Matplotlib, Seaborn, and Plotly. Our methodology employs natural-language instruction-driven code generation, coupled with standardized prompting and execution-based code evaluation. Key contributions include: (1) the first fine-grained, task-level evaluation framework for exploratory data visualization from tables; (2) empirical identification of a significant generalization gap for LLMs on Plotly—despite strong performance on Matplotlib and Seaborn; (3) demonstration of robustness to concise instructions across the latter two libraries; and (4) open-sourcing of both the benchmark dataset and evaluation framework, which has been widely adopted by the research community.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Web data visualization

Abstract

This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration. Our benchmark focuses on generating code for visualizing tabular data - such as a Pandas DataFrame - based on natural language instructions, complementing current evaluation tools and expanding their scope. The dataset includes 175 unique tasks. Our experiments assess several leading Large Language Models (LLMs) across three visualization libraries: Matplotlib, Seaborn, and Plotly. We show that the shortening of tasks has a minimal effect on plotting capabilities, allowing for the user interface that accommodates concise user input without sacrificing functionality or accuracy. Another of our findings reveals that while LLMs perform well with popular libraries like Matplotlib and Seaborn, challenges persist with Plotly, highlighting areas for improvement. We hope that the modular design of our benchmark will broaden the current studies on generating visualizations. Our dataset and benchmark code are available online: https://huggingface.co/datasets/JetBrains-Research/PandasPlotBench; https://github.com/JetBrains-Research/PandasPlotBench.

Problem

• Evaluates LLMs in generating data visualization code
• Focuses on code generation from natural language instructions
• Assesses performance across Matplotlib, Seaborn, and Plotly libraries

Innovation

• Human-curated PandasPlotBench dataset
• Assesses Large Language Models (LLMs)
• Focuses on Matplotlib, Seaborn, Plotly