How Do LLMs See Charts? A Comparative Study on High-Level Visualization Comprehension in Humans and LLMs

2026-04-10

AI Summary

This study investigates the discrepancies between large language models (LLMs) and human cognition in high-level chart understanding, with a focus on interpreting designer intent and extracting complex data patterns. Through qualitative user studies, it systematically compares the higher-order interpretation strategies employed by humans and LLMs on line charts, bar charts, and scatter plots, while analyzing LLM outputs and reasoning pathways under three distinct prompting conditions. The work reveals, for the first time, that LLMs consistently adopt a structured enumeration strategy rather than constructing coherent trend-based narratives, and their explanatory patterns remain remarkably stable across different prompts. In contrast, humans demonstrate a superior ability to synthesize holistic, narrative-driven interpretations. These findings highlight fundamental mechanistic limitations of LLMs in visual reasoning and offer critical insights for future model design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Designers often create visualizations to achieve specific high-level analytical or communication goals. These goals require people to extract complex and interconnected data patterns. Prior perceptual studies of visualization effectiveness have focused on low-level tasks, such as estimating statistical quantities, and have recently explored high-level comprehension of visualization. Despite the growing use of Large Language Models (LLMs) as visualization interpreters, how their interpretations relate to human understanding or what reasoning processes underlie their responses remains insufficiently understood. In this work, we explore LLMs'visualization comprehension, examining the alignment between designers'communicative goals and what their audience sees in a visualization. We have conducted a qualitative study to investigate the gap between human interpretative strategies and the reasoning pathways of LLMs across three types of visualizations, line graphs, bar graphs, and scatterplots, to identify the high-level patterns generated by LLMs using three prompt conditions. Our analysis results indicate that LLMs exhibit a consistent interpretative strategy that remains unchanged across prompt constraints. Furthermore, we observe two distinct approaches: humans naturally synthesize data into trend-centric narratives, whereas LLMs persist with a structural enumeration of comparisons and numerical ranges. Lastly, we see LLMs achieve visualization comprehension through mechanisms distinct from human intuition, pointing to critical challenges and new opportunities for visualization design.

Problem

• Large Language Models
• visualization comprehension
• human interpretation
• communicative goals
• high-level patterns

Innovation

• visualization comprehension
• Large Language Models
• human-AI comparison
• interpretative strategies
• prompt conditions