Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation

2026-05-02

AI Summary

This study investigates which prompt dimensions dominate the spatial attention mechanism of zero-shot vision-language models (VLMs) in non-small cell lung cancer tumor segmentation. Focusing on the VoxTell model, the authors systematically analyze its alignment with clinical prompts through sub-prompt decomposition, attribute perturbation, specificity-aware gradient analysis, and cross-case prompt swapping. They reveal, for the first time, that anatomical location is the primary factor governing VLM spatial attention and argue that model evaluation should incorporate prompt-alignment dimensions. Experimental results demonstrate that VoxTell achieves a mean Dice similarity coefficient of 0.613 under a fully zero-shot setting, showing no statistically significant difference compared to the fine-tuned nnUNet (Wilcoxon test with Benjamini–Hochberg correction) and substantially outperforming existing zero-shot approaches.

Abstract

Zero-shot vision-language models (VLMs) offer a promptable alternative to task-specific training for gross tumor volume (GTV) delineation in non-small-cell lung cancer (NSCLC), but the prompt dimensions that govern their spatial behavior remain poorly understood. We study this question by probing alignment directions in VoxTell on a held-out internal NSCLC tumor dataset through sub-prompt decomposition into diagnosis, demographic, staging, anatomical, generic, and irrelevant controls; attribute-wise perturbation robustness; specificity ladders; and cross-case prompt swaps, while benchmarking against fine-tuned and zero-shot baselines using the Dice Similarity Coefficient (DSC) with Wilcoxon signed-rank tests and Benjamini-Hochberg correction. Alignment analyses revealed that anatomical location is the dominant driver of VoxTell's spatial attention: 63.4 percent of location perturbations caused catastrophic drops, prompt specificity improved from generic to full descriptions except for diagnosis-only prompts, irrelevant prompts correctly yielded zero segmentation, and cross-case prompt swaps confirmed patient-specific conditioning (matched DSC 0.906 vs. mismatched 0.406). Histology and stage substitutions had minimal effect, indicating that the model prioritizes "where to look" over "what to look for." In this context, VoxTell, operating fully zero-shot, achieved a mean DSC of 0.613, statistically indistinguishable from nnUNet (0.690, adjusted p = 0.156) and Ahmed et al. (0.675, adjusted p = 0.679), while significantly outperforming all other zero-shot models. Together, these findings argue that segmentation VLMs should be evaluated not only by Dice, but also by the prompt dimensions to which they align.

Problem

• zero-shot segmentation
• vision-language models
• NSCLC tumor segmentation
• prompt alignment
• clinical factors

Innovation

• zero-shot segmentation
• vision-language models
• prompt alignment
• clinical factors
• NSCLC tumor segmentation