BIOCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models

2025-10-22

AI Summary

Biological image analysis suffers from scarce natural language descriptions and overreliance on coarse-grained class labels, limiting fine-grained semantic understanding. Method: We propose leveraging generative multimodal large language models (MLLMs) to synthesize instance-level descriptive captions as auxiliary supervision. Specifically, we fuse Wikipedia visual content with taxon-specific templates to automatically generate biologically meaningful, instance-level captions, which are then used to train BIOCAP—a biological multimodal foundation model optimized for image–text alignment. Contribution/Results: This work presents the first systematic validation of fine-grained textual supervision beyond class labels for biological visual representation learning. Experiments demonstrate that BIOCAP significantly outperforms baseline models on species classification and cross-modal retrieval tasks. The synthesized captions effectively enhance alignment between visual and semantic spaces, establishing a novel paradigm for multimodal modeling in data-scarce biological domains.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a species, each capturing certain biological traits. Incorporating captions during training encourages alignment with this shared latent structure, emphasizing potentially diagnostic characters while suppressing spurious correlations. The main challenge, however, lies in obtaining faithful, instance-specific captions at scale. This requirement has limited the utilization of natural language supervision in organismal biology compared with many other scientific domains. We complement this gap by generating synthetic captions with multimodal large language models (MLLMs), guided by Wikipedia-derived visual information and taxon-tailored format examples. These domain-specific contexts help reduce hallucination and yield accurate, instance-based descriptive captions. Using these captions, we train BIOCAP (i.e., BIOCLIP with Captions), a biological foundation model that captures rich semantics and achieves strong performance in species classification and text-image retrieval. These results demonstrate the value of descriptive captions beyond labels in bridging biological images with multimodal foundation models.

Problem

• Generating synthetic captions for biological images using multimodal language models
• Training biological foundation models with descriptive captions beyond simple labels
• Improving species classification and text-image retrieval through caption supervision

Innovation

• Generates synthetic captions using multimodal language models
• Trains biological foundation model with descriptive captions
• Uses Wikipedia and taxon examples to reduce hallucination