LiveXiv - A Multi-Modal Live Benchmark Based on Arxiv Papers Content

2024-10-14

arXiv.org

AI Summary

Test data contamination from web crawling undermines the validity of multimodal model evaluation. Method: This paper introduces the first dynamic, evolvable “living” multimodal benchmark grounded in arXiv scientific papers. It (1) achieves precise figure–text alignment via multimodal PDF/TeX parsing; (2) jointly employs rule-based heuristics and large language models (LLMs) to generate high-quality, human-annotation-free visual question answering (VQA) data; and (3) proposes an incremental benchmark evolution mechanism coupled with a statistically grounded sparse-subset evaluation algorithm for efficient full-benchmark performance estimation. Contribution/Results: Human validation confirms an automatic annotation error rate <2.5%. Comprehensive evaluation on the inaugural benchmark reveals significant, previously undetected capability gaps across leading open- and closed-source multimodal large models. The dataset is publicly available on Hugging Face; code will be released shortly.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effectively on multiple downstream tasks. However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated. To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs (VQA). This is done without any human-in-the-loop, using the multi-modal content in the manuscripts, like graphs, charts, and tables. Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities, avoiding contamination. Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance variance is indeed minimal (<2.5%). Our dataset is available online on HuggingFace, and our code will be available here.

Problem

• Prevent test data contamination in multi-modal models
• Automate VQA pair generation from ArXiv papers
• Reduce evaluation cost with efficient subset testing

Innovation

• Automated VQA pairs from ArXiv papers
• Efficient subset-based evaluation approach
• Minimal variance in automatic annotations