Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

2026-06-29

AI Summary

Existing multimodal large language models struggle to accurately extract numerical data from chart images lacking explicit labels, limiting the reproducibility and analytical reliability of visualizations. This work addresses this challenge by drawing inspiration from human cognitive processes in chart interpretation, introducing the first label-free benchmark based on real-world charts, and proposing a progressive training framework that mimics the staged manner in which humans comprehend visual data. By integrating a hybrid active learning workflow, the approach achieves state-of-the-art performance on a 7B-parameter model, substantially improving numerical recovery accuracy. User studies further confirm its practical utility for efficient and reliable chart data extraction.

Abstract

Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign. Existing interactive tools are reliable but tedious, and mixed-initiative systems, while more efficient, lack generalizability. Recent multimodal large language models (MLLMs) offer a unified interface for chart interpretation, yet their ability to extract accurate data tables, especially without visible labels, remains unclear. We build a benchmark featuring diverse real-world charts without data labels to evaluate this capability. Results show that, while current MLLMs reliably reconstruct table structures, they struggle with precise value recovery. To address this, we revisit chart data extraction from a human-centered perspective and argue that extraction should follow a progressive learning process similar to how people read charts. Our training framework substantially improves numerical accuracy, achieving state-of-the-art performance with a 7B-parameter model. A user study further shows that our model effectively supports mixed-initiative workflows for reliable chart data extraction.

Problem

• chart data extraction
• multimodal large language models
• data table reconstruction
• numerical accuracy
• reproducibility

Innovation

• multimodal LLMs
• chart data extraction
• progressive learning
• benchmark
• numerical accuracy