PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models

2026-06-23

AI Summary

This work addresses the lack of large-scale, diverse, and readily accessible benchmarks for evaluating the robustness of vision-language models (VLMs) under multimodal adversarial attacks. To this end, we present the first systematic integration and expansion of existing adversarial benchmarks into an open-source, large-scale multimodal adversarial dataset. It encompasses 10 high-level categories, 55 harmful intent subcategories, totaling 7,826 distinct intents and 47,524 high-quality samples. The data are generated using state-of-the-art multimodal attack strategies and rigorously validated through automated classification and human verification to ensure semantic consistency and attack efficacy. This dataset substantially enhances evaluation diversity and practicality, enabling robustness testing, defense validation, and fine-tuning of attack models, thereby advancing standardized and reproducible research on VLM safety.

Abstract

We introduce a large-scale, open-source dataset of pre-generated adversarial attacks for vision-language models (VLMs). The dataset is designed to be diverse, representative, and practical, extending existing benchmarks by covering 10 high-level categories and 55 subcategories of harmful intents. Our primary goal is to make adversarial data accessible to the research community, given the computational cost and complexity of generating large numbers of attacks. The dataset comprises 47 524 adversarial samples, generated using state-of-the-art attack strategies from recent literature. Our work complements existing efforts by consolidating and extending prior benchmarks from multiple established sources, resulting in 7 826 intents, and introduce an additional category to broaden coverage. This provides realistic evaluation resources for studying model robustness and alignment. Our dataset intends to enable researchers and practitioners to systematically evaluate the robustness and safety of VLMs, fine-tune attack-generation models, and develop or stress-test defensive guardrails under diverse adversarial conditions. By releasing this resource, we aim to lower the barrier to adversarial research and foster more reproducible, comprehensive, and comparable evaluations of VLM safety.

Problem

• adversarial attacks
• vision-language models
• multimodal
• robustness
• safety

Innovation

• multimodal adversarial attacks
• vision-language models
• large-scale dataset
• model robustness
• safety evaluation