Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction

2025-08-06

AI Summary

Large language models (LLMs) exhibit limited visualization literacy, hindering reliable chart interpretation. Method: We propose Charts-of-Thought (CoT), a structured prompting framework that guides LLMs through sequential steps—data extraction, cross-validation, and logical reasoning—to systematically enhance multimodal chart understanding. CoT is the first approach to embed interpretable, stepwise reasoning into multimodal visual inference. Contribution/Results: Evaluated using the Visualization Literacy Assessment Test (VLAT) on Claude-3.7, GPT-4.5, and Gemini-2.0, CoT achieves a VLAT score of 50.17 for Claude-3.7-sonnet—significantly surpassing the human baseline (28.82). Across models, average performance improves by 15.2%, with 100% accuracy attained on several chart types. This work establishes a reproducible, generalizable, structured analytical paradigm for visual understanding tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper evaluates the visualization literacy of modern Large Language Models (LLMs) and introduces a novel prompting technique called Charts-of-Thought. We tested three state-of-the-art LLMs (Claude-3.7-sonnet, GPT-4.5 preview, and Gemini-2.0-pro) on the Visualization Literacy Assessment Test (VLAT) using standard prompts and our structured approach. The Charts-of-Thought method guides LLMs through a systematic data extraction, verification, and analysis process before answering visualization questions. Our results show Claude-3.7-sonnet achieved a score of 50.17 using this method, far exceeding the human baseline of 28.82. This approach improved performance across all models, with score increases of 21.8% for GPT-4.5, 9.4% for Gemini-2.0, and 13.5% for Claude-3.7 compared to standard prompting. The performance gains were consistent across original and modified VLAT charts, with Claude correctly answering 100% of questions for several chart types that previously challenged LLMs. Our study reveals that modern multimodal LLMs can surpass human performance on visualization literacy tasks when given the proper analytical framework. These findings establish a new benchmark for LLM visualization literacy and demonstrate the importance of structured prompting strategies for complex visual interpretation tasks. Beyond improving LLM visualization literacy, Charts-of-Thought could also enhance the accessibility of visualizations, potentially benefiting individuals with visual impairments or lower visualization literacy.

Problem

• Evaluating LLM visualization literacy using VLAT
• Introducing Charts-of-Thought for structured data extraction
• Improving LLM performance on visual interpretation tasks

Innovation

• Structured prompting for data extraction
• Systematic verification and analysis process
• Enhancing LLM visualization literacy