Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models

2026-08-10

AI Summary

This work addresses the limited adversarial robustness of existing Vision-Language-Action (VLA) models in physical environments, where current attack methods often produce visible artifacts or require white-box access, hindering real-world deployment. To overcome these limitations, we propose DURA—the first unrestricted adversarial attack based on diffusion models—that optimizes within the latent trajectory space of a pretrained diffusion model to generate visually natural adversarial patches. Without relying on pixel-level perturbations, DURA effectively steers VLA models to execute attacker-specified actions. Our method supports both white-box and black-box settings—requiring only action predictions—and consistently outperforms existing approaches in both simulated and real robotic environments, thereby exposing critical safety vulnerabilities in practical VLA deployments.

Abstract

Vision-Language-Action (VLA) models have shown strong capabilities in controlling robots across diverse manipulation tasks. However, their adversarial robustness remains largely underexplored, and exploiting this weakness can lead to physical-world harm. Existing attacks on VLA models often rely on pixel-space perturbations or white-box access, resulting in noticeable artifacts and limited deployability in real-world robotic systems. In this work, we propose DURA, a diffusion-based unrestricted robotic attack that generates visually natural adversarial patches for VLA models. DURA supports both white-box and black-box attack settings, where the black-box setting requires only the predicted actions of the victim model. By optimizing along the latent trajectory of a pretrained diffusion model, DURA generates visually natural patches while steering the robot toward attacker-specified target actions. Extensive experiments in both simulation and the real physical world show that DURA consistently outperforms existing methods. Our findings expose a safety risk for physically deployed VLA models and call for stronger defenses.

Problem

• Vision-Language-Action models
• adversarial robustness
• unrestricted attacks
• physical-world harm
• visually natural adversarial patches

Innovation

• diffusion-based attack
• vision-language-action models
• adversarial patches
• black-box attack
• robotic manipulation