MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research

2025-03-17

AI Summary

Existing benchmarks either emphasize undergraduate-level difficulty or focus narrowly on low-level perception, failing to assess the complex multimodal reasoning required in scientific research. To address this gap, we introduce MicroVQA—the first visual question answering benchmark tailored for research-grade microscopic image understanding—centered on three core scientific competencies: expert-level image interpretation, hypothesis generation, and experimental proposal. It comprises 1,042 expert-annotated, multimodal multiple-choice questions grounded in biological domain knowledge. Our method features a novel two-stage MCQ generation pipeline: structured LLM prompting followed by RefineBot-based elimination of linguistic shortcuts. Evaluation is conducted within an MLLM framework integrating expert knowledge injection, scientific literature fine-tuning, and chain-of-thought analysis. State-of-the-art models achieve only 53% accuracy, primarily due to perceptual errors; notably, smaller models perform comparably to larger ones, indicating that multimodal alignment poses significantly greater challenges than pure language reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Community question answering
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Scientific research demands sophisticated reasoning over multimodal data, a challenge especially prevalent in biology. Despite recent advances in multimodal large language models (MLLMs) for AI-assisted research, existing multimodal reasoning benchmarks only target up to college-level difficulty, while research-level benchmarks emphasize lower-level perception, falling short of the complex multimodal reasoning needed for scientific discovery. To bridge this gap, we introduce MicroVQA, a visual-question answering (VQA) benchmark designed to assess three reasoning capabilities vital in research workflows: expert image understanding, hypothesis generation, and experiment proposal. MicroVQA consists of 1,042 multiple-choice questions (MCQs) curated by biology experts across diverse microscopy modalities, ensuring VQA samples represent real scientific practice. In constructing the benchmark, we find that standard MCQ generation methods induce language shortcuts, motivating a new two-stage pipeline: an optimized LLM prompt structures question-answer pairs into MCQs; then, an agent-based `RefineBot' updates them to remove shortcuts. Benchmarking on state-of-the-art MLLMs reveal a peak performance of 53%; models with smaller LLMs only slightly underperform top models, suggesting that language-based reasoning is less challenging than multimodal reasoning; and tuning with scientific articles enhances performance. Expert analysis of chain-of-thought responses shows that perception errors are the most frequent, followed by knowledge errors and then overgeneralization errors. These insights highlight the challenges in multimodal scientific reasoning, showing MicroVQA is a valuable resource advancing AI-driven biomedical research. MicroVQA is available at https://huggingface.co/datasets/jmhb/microvqa, and project page at https://jmhb0.github.io/microvqa.

Problem

• Addresses lack of research-level multimodal reasoning benchmarks in biology.
• Introduces MicroVQA to assess expert image understanding and hypothesis generation.
• Highlights challenges in multimodal scientific reasoning and perception errors.

Innovation

• Introduces MicroVQA for advanced multimodal reasoning.
• Uses two-stage pipeline for MCQ generation.
• Benchmarks MLLMs, highlights multimodal reasoning challenges.