Measuring and predicting variation in the difficulty of questions about data visualizations

2025-05-12

AI Summary

Prior research lacks a cognitively grounded, quantitative account of difficulty variation across data visualization comprehension tasks. Method: Leveraging large-scale assessment data from 503 adults, we administered five canonical visualization comprehension item types and employed multidimensional item response theory (MIRT), composite test design, and rigorous psychometric validation to obtain high-reliability, full-spectrum difficulty estimates. Contribution/Results: Traditional categorical predictors—e.g., chart type or task category—explain only modest variance in item difficulty (low R²), revealing fundamental limitations of coarse-grained classification approaches. To address this, we propose the first fine-grained, generalizable framework for predicting visualization comprehension difficulty, grounded in cognitive mechanisms rather than surface-level features. This framework yields empirically validated difficulty parameters, establishes a methodological benchmark for visualization literacy assessment, and provides novel insights into the cognitive architecture underlying visualization understanding.

Technology Category

• Data Mining & Knowledge Management: Data Visualization & Summarization
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Learning Preferences or Rankings

Application Category

• Web Mining and Content Analysis: Web data visualization
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Understanding what is communicated by data visualizations is a critical component of scientific literacy in the modern era. However, it remains unclear why some tasks involving data visualizations are more difficult than others. Here we administered a composite test composed of five widely used tests of data visualization literacy to a large sample of U.S. adults (N=503 participants).We found that items in the composite test spanned the full range of possible difficulty levels, and that our estimates of item-level difficulty were highly reliable. However, the type of data visualization shown and the type of task involved only explained a modest amount of variation in performance across items, relative to the reliability of the estimates we obtained. These results highlight the need for finer-grained ways of characterizing these items that predict the reliable variation in difficulty measured in this study, and that generalize to other tests of data visualization understanding.

Problem

• Measuring difficulty variation in data visualization questions
• Identifying factors affecting performance in visualization tasks
• Developing finer-grained methods to predict task difficulty

Innovation

• Composite test of five visualization literacy assessments
• Large sample analysis with 503 U.S. adults
• Fine-grained characterization for difficulty prediction