BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning

2025-08-13

AI Summary

Current vision-language models exhibit suboptimal performance in chart understanding, primarily due to insufficient realism and diversity in training data, reliance on noisy automatically extracted tabular representations, and limitations imposed by conventional single-stage supervised fine-tuning. To address these issues, we propose BigCharts: (1) a high-fidelity, diverse chart dataset integrating real-world charts with precise redrawing techniques; (2) a fine-grained reward function tailored for chart reasoning, coupled with the first application of Group Relative Policy Optimization (GRPO) to jointly optimize supervised fine-tuning and reinforcement learning for enhanced reasoning capability; and (3) integrated visual diversity augmentation strategies. Extensive experiments demonstrate that BigCharts achieves state-of-the-art performance across multiple chart question-answering benchmarks, significantly outperforming both leading open-source and proprietary large multimodal models.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Charts are essential to data analysis, transforming raw data into clear visual representations that support human decision-making. Although current vision-language models (VLMs) have made significant progress, they continue to struggle with chart comprehension due to training on datasets that lack diversity and real-world authenticity, or on automatically extracted underlying data tables of charts, which can contain numerous estimation errors. Furthermore, existing models only rely on supervised fine-tuning using these low-quality datasets, severely limiting their effectiveness. To address these issues, we first propose BigCharts, a dataset creation pipeline that generates visually diverse chart images by conditioning the rendering process on real-world charts sourced from multiple online platforms. Unlike purely synthetic datasets, BigCharts incorporates real-world data, ensuring authenticity and visual diversity, while still retaining accurate underlying data due to our proposed replotting process. Additionally, we introduce a comprehensive training framework that integrates supervised fine-tuning with Group Relative Policy Optimization (GRPO)-based reinforcement learning. By introducing novel reward signals specifically designed for chart reasoning, our approach enhances model robustness and generalization across diverse chart styles and domains, resulting in a state-of-the-art chart reasoning model, BigCharts-R1. Extensive experiments demonstrate that our models surpass existing methods on multiple chart question-answering benchmarks compared to even larger open-source and closed-source models.

Problem

• Improving chart comprehension in vision-language models
• Enhancing dataset diversity and real-world authenticity
• Integrating reinforcement learning for better chart reasoning

Innovation

• Generates diverse charts using real-world data
• Integrates supervised fine-tuning with GRPO reinforcement learning
• Introduces novel reward signals for chart reasoning