MedVLSynther: Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs

2025-10-29

AI Summary

Medical visual question answering (VQA) has long suffered from a scarcity of high-quality, openly accessible training data. To address this, we propose the first scoring-criterion-guided, generate-then-verify two-stage framework for automatically synthesizing clinically accurate, modality-consistent multiple-choice VQA data from PubMed Central articles. Our method employs a large multimodal model as the generator, performing joint reasoning over medical images, figure captions, and corresponding textual content. A multi-level verifier—incorporating domain-specific consistency checks and a verifiable reward mechanism—enables reinforcement learning optimization and produces machine-auditable JSON outputs. We introduce MedSynVQA, a novel dataset comprising 13,087 questions spanning 13 imaging modalities and 28 anatomical regions. Evaluated on six medical VQA benchmarks, our approach achieves state-of-the-art performance, notably attaining 77.57 on VQA-RAD—surpassing leading medical foundation models.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Multimodal Models (LMMs) are increasingly capable of answering medical questions that require joint reasoning over images and text, yet training general medical VQA systems is impeded by the lack of large, openly usable, high-quality corpora. We present MedVLSynther, a rubric-guided generator-verifier framework that synthesizes high-quality multiple-choice VQA items directly from open biomedical literature by conditioning on figures, captions, and in-text references. The generator produces self-contained stems and parallel, mutually exclusive options under a machine-checkable JSON schema; a multi-stage verifier enforces essential gates (self-containment, single correct answer, clinical validity, image-text consistency), awards fine-grained positive points, and penalizes common failure modes before acceptance. Applying this pipeline to PubMed Central yields MedSynVQA: 13,087 audited questions over 14,803 images spanning 13 imaging modalities and 28 anatomical regions. Training open-weight LMMs with reinforcement learning using verifiable rewards improves accuracy across six medical VQA benchmarks, achieving averages of 55.85 (3B) and 58.15 (7B), with up to 77.57 on VQA-RAD and 67.76 on PathVQA, outperforming strong medical LMMs. A Ablations verify that both generation and verification are necessary and that more verified data consistently helps, and a targeted contamination analysis detects no leakage from evaluation suites. By operating entirely on open literature and open-weight models, MedVLSynther offers an auditable, reproducible, and privacy-preserving path to scalable medical VQA training data.

Problem

• Synthesizing medical VQA data from open literature using generator-verifier LMMs
• Addressing lack of high-quality training corpora for medical VQA systems
• Improving medical VQA accuracy through verifiable reward-based training

Innovation

• Generator-verifier framework synthesizes VQA from biomedical documents
• Multi-stage verifier enforces clinical validity and image-text consistency
• Training LMMs with verifiable rewards improves medical VQA accuracy