XRF-to-Optical Field-of-View Localization with Vision Language Models

2026-08-18

AI Summary

研究使用视觉语言模型解决XRF与光学显微图像视场定位问题，通过无训练模型和经典模板匹配等方法，在低对应图像中实现有效定位。

Abstract

Registering images acquired with different microscopy modalities is essential for relating complementary measurements of the same specimen. In correlative X-ray fluorescence (XRF) and optical microscopy, the XRF map often covers only a small region of an optical image acquired from the same or an adjacent tissue section. Field-of-view (FOV) localization is necessary but can be difficult when appearance and structure differ across modalities. Here we evaluate training-free vision language model (VLM) localization on two datasets representing same-section high-correspondence and adjacent-section low-correspondence imaging. We test unconstrained and metadata-constrained search and compare VLMs with geometric controls, classical template matching, and two alternative training-free approaches (DINOv2 and multiGradICON). Direct VLM prompting produced content-dependent spatial signals but was not reliable alone. Classical matching was most accurate when cross-modal structure was preserved but failed in the low-correspondence collection. A proposal-and-verify workflow used repeated VLM predictions as candidates and image-based similarity to select the final location. This workflow recovered useful localization in the low-correspondence regime.

Problem

• XRF
• optical microscopy
• field-of-view localization
• multimodal imaging
• correlative microscopy

Innovation

• vision language model
• field-of-view localization
• correlative microscopy
• proposal-and-verify workflow
• cross-modal structure