In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopy

2025-11-04

AI Summary

Cell detection in optical microscopy images suffers from scarce annotated data and poor cross-domain generalization. Method: We propose a novel few-shot object detection paradigm for biomedical microscopic imaging. Specifically, we introduce Micro-OD—the first few-shot detection benchmark for microscopy images—and design a hybrid pipeline integrating a standard detection head with a vision-language model (VLM) classifier, enhanced by an implicit reasoning token mechanism to improve end-to-end localization. We systematically evaluate multiple VLMs under in-context learning for few-shot generalization. Contribution/Results: We reveal the critical role of reasoning tokens in precise localization, demonstrating performance saturation with only six support examples. Moreover, we provide the first reproducible, open-vocabulary microscopy detection platform, significantly improving VLMs’ few-shot adaptability and detection accuracy on low-resource biomedical imagery.

Technology Category

• Computer Vision: Medical and Biological Imaging
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Foundation vision-language models (VLMs) excel on natural images, but their utility for biomedical microscopy remains underexplored. In this paper, we investigate how in-context learning enables state-of-the-art VLMs to perform few-shot object detection when large annotated datasets are unavailable, as is often the case with microscopic images. We introduce the Micro-OD benchmark, a curated collection of 252 images specifically curated for in-context learning, with bounding-box annotations spanning 11 cell types across four sources, including two in-lab expert-annotated sets. We systematically evaluate eight VLMs under few-shot conditions and compare variants with and without implicit test-time reasoning tokens. We further implement a hybrid Few-Shot Object Detection (FSOD) pipeline that combines a detection head with a VLM-based few-shot classifier, which enhances the few-shot performance of recent VLMs on our benchmark. Across datasets, we observe that zero-shot performance is weak due to the domain gap; however, few-shot support consistently improves detection, with marginal gains achieved after six shots. We observe that models with reasoning tokens are more effective for end-to-end localization, whereas simpler variants are more suitable for classifying pre-localized crops. Our results highlight in-context adaptation as a practical path for microscopy, and our benchmark provides a reproducible testbed for advancing open-vocabulary detection in biomedical imaging.

Problem

• Adapting vision-language models for few-shot cell detection in microscopy
• Addressing domain gap between natural images and biomedical microscopy data
• Enhancing object detection when large annotated datasets are unavailable

Innovation

• In-context learning for few-shot cell detection
• Hybrid pipeline combining detection head with VLM classifier
• Micro-OD benchmark with 252 annotated microscopy images