Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation

2026-04-27

AI Summary

This work addresses the performance bias of multimodal language models (MLMs) in chart-to-table conversion tasks, stemming from the uneven distribution of Y-axis characteristics in existing public chart datasets. The authors propose FairChart2Table, a framework that systematically evaluates five state-of-the-art MLMs with respect to key Y-axis attributes—namely tick label length, count, numerical range, and formatting—and further investigates the impact of legend count and prompt design. Experimental results demonstrate that Y-axis features significantly influence model accuracy, with legend count exhibiting a negative correlation with performance. Moreover, incorporating Y-axis–aware prompts effectively mitigates such biases and improves table generation accuracy for several models.

Abstract

Chart-to-table translation converts chart images into structured tabular data. Accurate translation is crucial for Multimodal Language Model (MLM) to answer complex queries. We observe imbalances in the number of images across different aspects of the y-axis information in public chart datasets. Such imbalances can introduce unintended biases, causing uneven MLM performance. Previous works have not systematically examined these biases. To address this gap, we propose a new framework, FairChart2Table, for analyzing y-axis-related bias on five state-of-the-art models. Key Findings: (1) There are significant y-axis biases related to the digit length of the major tick values, the number of major ticks, the range of values, and the tick value format (e.g., abbreviation or scientific format). (2) The number of legends/entities in chart images impacts MLM performance. (3) Prompting MLM with y-axis information can significantly enhance the performance for some MLMs.

Problem

• y-axis bias
• chart-to-table translation
• multimodal language models
• dataset imbalance
• structured data extraction

Innovation

• y-axis bias
• chart-to-table translation
• multimodal language models
• FairChart2Table
• prompting strategy