Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models

2026-06-24

AI Summary

Current adversarial research on text, vision, and vision-language models is fragmented across four independent tracks, lacking a unified framework and standardized evaluation protocols. This work proposes the first systematic integrative framework for cross-modal adversarial studies, with a focus on large language models, establishing a cohesive conceptual taxonomy encompassing attacks, defenses, and evaluation. The study innovatively introduces a six-category role classification for diffusion models, integrates a threat-model axis with a five-dimensional evaluation metric, and adopts denoising diffusion models as the core generation mechanism. Through a systematic review of 50 papers and 14 baseline methods, the authors identify five critical weaknesses in existing research and outline key open problems alongside a roadmap for future experimentation.

Abstract

Adversarial evaluation of AI systems has matured along four largely disconnected tracks: diffusion-based attacks on text and large language models (LLMs), diffusion-based attacks on image classifiers, jailbreak pipelines against vision-language models, and diffusion-based input purification defenses. Each has developed its own vocabulary, threat models, and benchmarks, with denoising diffusion models emerging as a shared generative mechanism whose recipes are now actively ported between communities. This survey performs an information-fusion exercise at the meta-research level: we integrate these four tracks into a single conceptual framework with a unified taxonomy, evaluation criteria, and research agenda, focusing on the LLM-side slice. We catalog fifty published papers across four scope areas (text/LLM, image classifier, vision-language model, defense), plus four diffusion-LLM-as-victim entries and ten non-diffusion baselines against which any new attack must be compared. We propose a six-class taxonomy of diffusion roles in adversarial pipelines, augmented by a threat-model axis recording attacker knowledge, query budget, and target accessibility, and apply a five-dimension framework (attack success rate, transferability, query budget, perplexity, defense-evasion) uniformly across modalities. The review adopts a dual attacker-defender perspective: alongside the attack catalog we cover four diffusion-based defenses that form the natural evaluation backdrop for new attacks. Our critical analysis identifies five recurring weaknesses of the current LLM-side literature, and we close with a research agenda of open questions and concrete experimental designs. The companion catalog and spreadsheet are released with the paper. We are explicit that this is a narrative review with quality assessment, not a PRISMA-compliant systematic review, and discuss the implications for replication.

Problem

• adversarial evaluation
• diffusion models
• multimodal AI
• large language models
• vision-language models

Innovation

• adversarial diffusion
• unified taxonomy
• cross-modal evaluation
• diffusion-based defense
• LLM jailbreak