BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature

2025-01-13

AI Summary

Existing biomedical multimodal datasets are scarce and lack interpretability for non-expert users—such as middle-school students—hindering the deployment of vision-language models (VLMs) in education and clinical settings. Method: We introduce BMCA, the first large-scale, open biomedical image–text pair archive, comprising over 24 million semantically aligned image–caption pairs automatically extracted from PubMed Central. We propose an end-to-end, scientific-literature-driven multimodal data construction framework integrating OCR, document layout analysis, and expert-guided meta-annotation. We further release the streaming-pretrained BMCA-CLIP model family, eliminating the need to download 27 TB of raw data. Results: BMCA-CLIP achieves state-of-the-art performance across 40 cross-disciplinary biomedical tasks: zero-shot classification improves by +6.56% on average (+29.8% in dermatology, +17.5% in ophthalmology); image–text retrieval accuracy increases significantly; and computational overhead is reduced tenfold.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

The development of vision-language models (VLMs) is driven by large-scale and diverse multimodal datasets. However, progress toward generalist biomedical VLMs is limited by the lack of annotated, publicly accessible datasets across biology and medicine. Existing efforts are restricted to narrow domains, missing the full diversity of biomedical knowledge encoded in scientific literature. To address this gap, we introduce BIOMEDICA, a scalable, open-source framework to extract, annotate, and serialize the entirety of the PubMed Central Open Access subset into an easy-to-use, publicly accessible dataset.Our framework produces a comprehensive archive with over 24 million unique image-text pairs from over 6 million articles. Metadata and expert-guided annotations are also provided. We demonstrate the utility and accessibility of our resource by releasing BMCA-CLIP, a suite of CLIP-style models continuously pre-trained on the BIOMEDICA dataset via streaming, eliminating the need to download 27 TB of data locally.On average, our models achieve state-of-the-art performance across 40 tasks - spanning pathology, radiology, ophthalmology, dermatology, surgery, molecular biology, parasitology, and cell biology - excelling in zero-shot classification with a 6.56% average improvement (as high as 29.8% and 17.5% in dermatology and ophthalmology, respectively), and stronger image-text retrieval, all while using 10x less compute. To foster reproducibility and collaboration, we release our codebase and dataset for the broader research community.

Problem

• Biomedical Dataset
• Visual Language Models
• Educational Content

Innovation

• BIOMEDICA
• BMCA-CLIP
• Medical Image Recognition