Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks

2026-04-06

AI Summary

Existing vision-language models lack systematic evaluation of their robustness to natural semantic variations beyond standard benchmarks. This work proposes the first comprehensive evaluation framework across diverse downstream tasks—including zero-shot image classification, semantic segmentation, and visual question answering—to systematically audit the vulnerabilities of prominent models such as CLIP, Robust CLIP, BLIP-2, and SigLIP under typographic attacks, ImageNet-A, and language-induced adversarial examples. The study reveals that robust training does not universally enhance performance and can even degrade it in certain natural adversarial settings. Furthermore, all CLIP variants exhibit significant failure under language-induced perturbations, exposing critical failure modes and fundamental limitations in the robustness of current vision-language models.

Abstract

Recent advances in vision-language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.

Problem

• vision-language models
• robustness
• natural adversarial examples
• semantic variation
• systematic evaluation

Innovation

• vision-language models
• natural adversarial evaluation
• semantic variation
• robustness audit
• zero-shot transfer