Semantic Robustness Certification for Vision-Language Models

2026-06-17

AI Summary

Existing vision-language models lack verifiable robustness guarantees under distribution shifts induced by semantic variations such as shape, size, or style. This work proposes the first framework capable of certifying robustness against semantic-level perturbations without requiring additional data. Leveraging the open-vocabulary capacity of vision-language models, the method employs textual prompts as semantic proxies to construct controllable transformations and derives closed-form characterizations of decision boundaries to quantitatively certify invariant prediction regions under semantic perturbations. By moving beyond the conventional limitations of geometric or pixel-level perturbations, the approach demonstrates effective and practical robustness certification across diverse semantic transformations on both synthetic and real-world datasets.

Abstract

Vision-language models (VLMs) are now widely used in downstream tasks. However, real-world applications often expose VLMs to distribution shifts induced by semantic variation (e.g., shape, size, and style). Robustness certification determines if a model's prediction changes when transformations are applied to its input. While most certification frameworks study geometric or pixel-level transformations over inputs, this work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations. Leveraging the open-vocabulary capability of VLMs, we use text prompts as semantic proxies to construct transformations parameterized by an extent that controls the degree of semantic variation. By characterizing the VLM decision boundary in closed form, our framework quantitatively certifies extent intervals for which the predicted class remains unchanged under the semantic transformation. Our framework is the first to certify VLM robustness under semantic-level variations without requiring additional data for each variation, making it practical to apply. Experiments on both synthetic and real-world data show that our framework enables certifying robustness under diverse semantic variations across scenarios.

Problem

• semantic robustness
• vision-language models
• distribution shifts
• robustness certification
• semantic variation

Innovation

• semantic robustness certification
• vision-language models
• text prompts as semantic proxies
• decision boundary characterization
• open-vocabulary capability