LLMs have Visualization Literacy: Now What? Experiments Exploring LLM Visualization Evaluation Capabilities

2026-06-13

AI Summary

This study investigates whether large language models (LLMs) possess the core competencies required to serve as trustworthy and fair evaluators of data visualizations—namely, visualization literacy, instruction following, and preservation of graphical integrity. For the first time, an enhanced version of the Visualization Literacy Assessment Tool (VLAT) is integrated with assessments of the latter two dimensions to systematically evaluate prominent models, including Claude Opus 4.5, GPT 5.2 Pro, and Gemini 3 Flash. Results indicate that LLMs now surpass average human performance in visualization literacy; however, they struggle to reliably detect misleading charts without explicit prompting. Moreover, specialized prompting strategies such as few-shot examples and chain-of-thought reasoning yield only marginal improvements in their evaluation capabilities. This work provides empirical evidence of both the promise and limitations of deploying LLMs for visualization assessment tasks.

Abstract

As Large Language Models (LLMs) become more popular within the visualization community, researchers increasingly leverage them for diverse visualization tasks such as design guideline suggestions and visualization evaluation. However, in order for LLMs to act as trustworthy and fair evaluators, we argue that LLMs would need to possess visualization literacy, be capable of following user instructions and uphold graphical integrity. We test the latest versions of the most prominent LLMs, specifically Anthropic's Claude (Opus 4.5), OpenAI's Generative Pretrained Transformers (GPT 5.2 Pro), and Google's Gemini (Gemini 3 Flash) on these features and find that while these models now possess visualization literacy, they still struggle with other features necessary for instruction following and graphical integrity. Using a modified Visualization Literacy Assessment Test (VLAT), our findings show that these recent LLMs have achieved greater than human-levels of visualization literacy in contrast to prior research. In order to test the models' abilities to follow instructions, we used few-shot and chain-of-thought prompting as proxies for instruction following tasks on evaluating visualization literacy and find that these specialized prompting techniques are becoming obsolete with respect to improving visualization literacy. Additionally, we experiment with the inherent ability of LLMs to evaluate misleading visualizations to test the models' abilities for upholding graphical integrity and find that without specialized or leading prompting techniques, the models struggle with being able to accurately identify whether a visualization is misleading or not. Our results further break down the performance of each model on these tasks, but the culmination of our findings force us to reconsider the current effectiveness of LLMs as visualization evaluators.

Problem

• visualization literacy
• instruction following
• graphical integrity
• LLM evaluation
• misleading visualizations

Innovation

• visualization literacy
• instruction following
• graphical integrity
• large language models
• misleading visualizations