EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding

2024-09-03

arXiv.org

AI Summary

Real-world chart understanding is hindered by the scarcity of high-quality training data and the absence of realistic, comprehensive evaluation benchmarks. To address these challenges, we propose EvoChart—a novel self-training framework specifically designed for real-world chart understanding. EvoChart jointly constructs training data and a high-performance model via vision-language model (VLM)-driven synthetic chart generation and multi-source collection of authentic web charts. Concurrently, we introduce EvoChart-QA, the first large-scale, expert-annotated benchmark tailored to real-world chart comprehension, comprising 1,250 QA pairs drawn from 140 diverse websites. Experimental results on EvoChart-QA reveal that the strongest proprietary model, GPT-4o, achieves only 49.8% accuracy, whereas EvoChart boosts the accuracy of open-source VLMs to 54.2%, significantly outperforming baseline methods. These findings empirically validate the efficacy of both the self-training paradigm and domain-specific benchmarking in advancing chart understanding capabilities.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Large Vision Models
• Search and Optimization: Evolutionary Computation

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Chart understanding enables automated data analysis for humans, which requires models to achieve highly accurate visual comprehension. While existing Visual Language Models (VLMs) have shown progress in chart understanding, the lack of high-quality training data and comprehensive evaluation benchmarks hinders VLM chart comprehension. In this paper, we introduce EvoChart, a novel self-training method for generating synthetic chart data to enhance VLMs' capabilities in real-world chart comprehension. We also propose EvoChart-QA, a noval benchmark for measuring models' chart comprehension abilities in real-world scenarios. Specifically, EvoChart is a unique self-training data synthesis approach that simultaneously produces high-quality training corpus and a high-performance chart understanding model. EvoChart-QA consists of 650 distinct real-world charts collected from 140 different websites and 1,250 expert-curated questions that focus on chart understanding. Experimental results on various open-source and proprietary VLMs tested on EvoChart-QA demonstrate that even the best proprietary model, GPT-4o, achieves only 49.8% accuracy. Moreover, the EvoChart method significantly boosts the performance of open-source VLMs on real-world chart understanding tasks, achieving 54.2% accuracy on EvoChart-QA.

Problem

• Chart Recognition
• Training Data Quality
• Evaluation Standards

Innovation

• EvoChart
• data generation
• chart understanding