Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping

2024-10-11

arXiv.org

AI Summary

Existing LVLM evaluation benchmarks suffer from static design and data contamination, failing to reflect models’ true capabilities. To address this, we propose VLB—a dynamic multimodal evaluation protocol. VLB employs a vision-language bootstrapping module to generate image-text co-evolving VQA samples in real time, coupled with a cross-modal consistency discrimination module to ensure semantic plausibility. It further introduces controllable joint perturbations and dynamic strategy composition, enabling complexity-adjustable and benchmark-evolvable generative evaluation. VLB establishes the first evaluation framework that co-evolves with LVLM capabilities. On SEEDBench, MMBench, and MME, it significantly mitigates data contamination and more accurately exposes model performance limits and reasoning deficiencies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across multimodal tasks such as visual perception and reasoning, leading to good performance on various multimodal evaluation benchmarks. However, these benchmarks keep a static nature and overlap with the pre-training data, resulting in fixed complexity constraints and data contamination issues. This raises the concern regarding the validity of the evaluation. To address these two challenges, we introduce a dynamic multimodal evaluation protocol called Vision-Language Bootstrapping (VLB). VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. To this end, VLB dynamically generates new visual question-answering samples through a multimodal bootstrapping module that modifies both images and language, while ensuring that newly generated samples remain consistent with the original ones by a judge module. By composing various bootstrapping strategies, VLB offers dynamic variants of existing benchmarks with diverse complexities, enabling the evaluation to co-evolve with the ever-evolving capabilities of LVLMs. Extensive experimental results across multiple benchmarks, including SEEDBench, MMBench, and MME, show that VLB significantly reduces data contamination and exposes performance limitations of LVLMs.

Problem

• Static benchmarks limit evaluation validity for LVLMs
• Data contamination issues from overlapping pretraining datasets
• Need dynamic complexity to match evolving LVLM capabilities

Innovation

• Dynamic multimodal evaluation with flexible complexity
• Vision-Language Bootstrapping for robust assessment
• Multimodal bootstrapping module modifies images and language