Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection

2026-04-18

AI Summary

This study investigates whether semantically equivalent but lexically distinct textual prompts lead to inconsistent visual-language grounding outcomes. Building upon DETR for region proposal generation and CLIP for language-conditioned selection, we systematically evaluate the impact of synonymous prompts on object localization using the COCO dataset. Our analysis reveals that, on average, each object yields 2.11 distinct localization results across six paraphrased prompts. Prompt ensembling fails to improve performance, and textual embedding similarity accounts for only 34% of the observed prediction discrepancies. These findings demonstrate that localization instability primarily stems from the argmax selection mechanism rather than distances in textual embedding space, thereby providing the first quantitative characterization of the structural and directional nature of prompt sensitivity in vision-language grounding.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision
• Search and Optimization: Local Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-language models enable open-vocabulary object grounding through natural language queries, under the implicit assumption that semantically equivalent descriptions yield consistent outputs. We examine this assumption using a controlled pipeline combining DETR for object proposals with CLIP for language-conditioned selection on 263 COCO val2017 images. We find that overlapping prompts such as "a person," "a human," and "a pedestrian" frequently select different instances, with mean instability of 2.11 distinct selections across six prompts. PCA analysis shows this variability is structured and directional, not random. Prompt ensembling does not improve quality and often shifts selections toward generic regions. We further show that text embedding proximity explains only 34% of grounding disagreement (r = -0.58), confirming that instability arises from the argmax selection mechanism rather than text-level distances alone.

Problem

• prompt sensitivity
• vision-language grounding
• object detection
• open-vocabulary
• language-conditioned selection

Innovation

• prompt sensitivity
• vision-language grounding
• object detection instability
• CLIP-DETR pipeline
• argmax selection bias