FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models

2025-07-20

AI Summary

Existing large vision-language models (LVLMs) exhibit poor performance in financial chart understanding, particularly struggling with complex temporal structures and domain-specific terminology. Method: We introduce FinChart-Bench—the first benchmark tailored to real-world financial scenarios—comprising 1,200 chart images and 7,016 human-annotated questions spanning true/false, multiple-choice, and open-ended formats. It systematically evaluates 25 open- and closed-source LVLMs, uniquely emphasizing temporal modeling, instruction following, and spatial reasoning—capabilities critical for financial chart interpretation. Contribution/Results: FinChart-Bench reveals fundamental limitations in current LVLMs’ reliability for automated financial analysis. Experiments show a narrowing performance gap between open- and closed-source models; however, several newly released models suffer from regression. Widespread deficiencies persist—including weak spatial reasoning and poor instruction comprehension—highlighting urgent needs for domain-adapted architectural and training innovations.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large vision-language models (LVLMs) have made significant progress in chart understanding. However, financial charts, characterized by complex temporal structures and domain-specific terminology, remain notably underexplored. We introduce FinChart-Bench, the first benchmark specifically focused on real-world financial charts. FinChart-Bench comprises 1,200 financial chart images collected from 2015 to 2024, each annotated with True/False (TF), Multiple Choice (MC), and Question Answering (QA) questions, totaling 7,016 questions. We conduct a comprehensive evaluation of 25 state-of-the-art LVLMs on FinChart-Bench. Our evaluation reveals critical insights: (1) the performance gap between open-source and closed-source models is narrowing, (2) performance degradation occurs in upgraded models within families, (3) many models struggle with instruction following, (4) both advanced models show significant limitations in spatial reasoning abilities, and (5) current LVLMs are not reliable enough to serve as automated evaluators. These findings highlight important limitations in current LVLM capabilities for financial chart understanding. The FinChart-Bench dataset is available at https://huggingface.co/datasets/Tizzzzy/FinChart-Bench.

Problem

• Evaluating LVLMs on financial chart comprehension
• Assessing performance gaps in financial chart understanding
• Identifying limitations in spatial reasoning of LVLMs

Innovation

• First benchmark for financial charts
• Evaluates 25 vision-language models
• Includes 1,200 annotated financial charts