Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning

2025-01-22

AI Summary

Current multimodal large language models (MLLMs) suffer from fixed image resolution constraints and inadequate numerical reasoning capabilities for scientific table understanding. To address these limitations, we propose the first dedicated multimodal understanding and reasoning framework for scientific tables. Our method introduces a dynamic-resolution image encoding mechanism enabling adaptive input handling; constructs a comprehensive, quality-centric scientific table resource comprising three components—MMSci-Pre (52K samples for structural recognition), MMSci-Ins (12K instruction-following examples), and MMSci-Eval (3,114 samples for numerical reasoning evaluation); and integrates domain-adaptive instruction tuning with numerically aware reasoning modeling. Extensive experiments demonstrate that our approach significantly outperforms general-purpose table baselines on both scientific table comprehension and numerical reasoning tasks, while exhibiting strong cross-domain generalization. All code and datasets are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent large language models (LLMs) have advanced table understanding capabilities but rely on converting tables into text sequences. While multimodal large language models (MLLMs) enable direct visual processing, they face limitations in handling scientific tables due to fixed input image resolutions and insufficient numerical reasoning capabilities. We present a comprehensive framework for multimodal scientific table understanding and reasoning with dynamic input image resolutions. Our framework consists of three key components: (1) MMSci-Pre, a domain-specific table structure learning dataset of 52K scientific table structure recognition samples, (2) MMSci-Ins, an instruction tuning dataset with 12K samples across three table-based tasks, and (3) MMSci-Eval, a benchmark with 3,114 testing samples specifically designed to evaluate numerical reasoning capabilities. Extensive experiments demonstrate that our domain-specific approach with 52K scientific table images achieves superior performance compared to 150K general-domain tables, highlighting the importance of data quality over quantity. Our proposed table-based MLLMs with dynamic input resolutions show significant improvements in both general table understanding and numerical reasoning capabilities, with strong generalisation to held-out datasets. Our code and data are publicly available at https://github.com/Bernard-Yang/MMSci_Table.

Problem

• Benchmarking multimodal scientific table understanding and reasoning
• Improving numerical reasoning capabilities for scientific tables
• Addressing fixed input resolution limitations in MLLMs

Innovation

• Dynamic input image resolutions for tables
• Domain-specific scientific table structure dataset
• Instruction tuning for numerical reasoning tasks