SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

2024-07-12

arXiv.org

AI Summary

Existing multimodal question-answering (QA) benchmarks for scientific papers are scarce, especially those tailored to middle-school students’ cognitive levels and emphasizing integrated text-image comprehension. This work introduces SPIQA—the first large-scale multimodal QA dataset for scientific papers, comprising 270K high-quality questions grounded in diverse visual elements (e.g., charts, tables, diagrams) and spanning interdisciplinary research literature. Methodologically, we propose a hybrid annotation paradigm combining multi-stage prompt-engineered automatic labeling with human refinement, supported by a retrieval-augmented chain-of-thought (CoT) evaluation framework powered by multimodal large language models (MLLMs). This enables fine-grained performance analysis across reasoning dimensions. We conduct systematic evaluation on 12 state-of-the-art MLLMs, demonstrating that context-aware retrieval and incorporation of auxiliary textual information significantly elevate model reasoning ceilings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Seeking answers to questions within long scientific research articles is a crucial area of study that aids readers in quickly addressing their inquiries. However, existing question-answering (QA) datasets based on scientific papers are limited in scale and focus solely on textual content. We introduce SPIQA (Scientific Paper Image Question Answering), the first large-scale QA dataset specifically designed to interpret complex figures and tables within the context of scientific research articles across various domains of computer science. Leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures, we employ automatic and manual curation to create the dataset. We craft an information-seeking task on interleaved images and text that involves multiple images covering plots, charts, tables, schematic diagrams, and result visualizations. SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits. Through extensive experiments with 12 prominent foundational models, we evaluate the ability of current multimodal systems to comprehend the nuanced aspects of research articles. Additionally, we propose a Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that allows fine-grained, step-by-step assessment and improves model performance. We further explore the upper bounds of performance enhancement with additional textual information, highlighting its promising potential for future research and the dataset's impact on revolutionizing how we interact with scientific literature.

Problem

• Scientific Paper Understanding
• Image-Text Question Answering
• Middle School Level Dataset

Innovation

• SPIQA
• Multimodal Scientific Content Understanding
• Integrated Information Retrieval