Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts

2026-04-23

AI Summary

This work addresses the lack of systematic research on joint understanding of multiple charts, which hinders complex cross-chart reasoning. We introduce PolyChartQA, the first medium-scale multi-chart question answering benchmark tailored for scientific literature, comprising 534 chart groups and 2,694 human-annotated question-answer pairs that span diverse question types and chart structures. Evaluation using state-of-the-art multimodal language models reveals a significant 27.4% drop in accuracy on these human-authored questions compared to standard benchmarks. To mitigate this gap, we propose targeted prompting strategies that improve L-Accuracy by 5.39%, highlighting both the limitations of current models in authentic multi-chart comprehension and their potential for improvement through tailored interventions.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Charts are widely used to present complex information. Deriving meaningful insights in real-world contexts often requires interpreting multiple related charts together. Research on understanding multi-chart images has not been extensively explored. We introduce PolyChartQA, a mid-scale dataset specifically designed for question answering over multi-chart images. PolyChartQA comprises 534 multi-chart images (with a total of 2,297 sub-charts) sourced from peer-reviewed computer science research publications and 2,694 QA pairs. We evaluate the performance of nine state-of-the-art Multimodal Language Models (MLMs) on PolyChartQA across question type, difficulty, question source, and key structural characteristics of multi-charts. Our results show a 27.4% LLM-based accuracy (L-Accuracy) drop on human-authored questions compared to MLM-generated questions, and a 5.39% L-accuracy gain with our proposed prompting method.

Problem

• multi-chart understanding
• question answering
• chart interpretation
• multimodal benchmark
• data visualization

Innovation

• multi-chart understanding
• visual question answering
• multimodal language models
• benchmark dataset
• prompting method