Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Robotics

2026-06-02

AI Summary

This work addresses the challenge of deploying adversarial attacks against vision-language-action (VLA) robotic systems in real-world settings, where full execution trajectories are typically unavailable. The authors propose a novel threat model based on partial observability—requiring only a prefix of the trajectory—and introduce a two-stage framework to generate fixed adversarial patches that disrupt both semantic understanding and action planning in VLA models. Key innovations include attention-map-guided patch placement and a joint perturbation strategy that simultaneously corrupts semantic grounding and amplifies trajectory curvature. To the best of the authors’ knowledge, this is the first method to achieve long-horizon adversarial attacks on VLA agents under partial observability, demonstrating significant reductions in task success rates and strong robustness across both simulated and real-world environments.

Abstract

Vision-language-action (VLA) models are gaining attention in robotics, yet their robustness to adversarial attacks remains largely unexplored. Existing work shows that adversarial patches can mislead VLA-based robots but assumes full access to the entire execution trajectory, an unrealistic requirement in practice. We address this limitation by formulating a partially observable threat model, where the adversary can exploit only a short prefix of the trajectory to generate a fixed patch applied to all subsequent frames. Under this setting, we propose a two-phase framework. First, we localize the patch using the model's attention maps to identify visually critical regions that correspond to the full instruction. Then, we optimize the patch to disrupt the semantic grounding of target objects and increase the curvature of action trajectories, thereby compounding failures in both perception and control. Extensive experiments in simulation and real-world robotic environments show that our method sustains adversarial effects under partial observability, inducing long-horizon disruptions and significantly reducing task success rates.

Problem

• partially observable
• adversarial patch attacks
• vision-language-action models
• robotics
• trajectory

Innovation

• partially observable attacks
• adversarial patch
• vision-language-action models
• semantic grounding disruption
• trajectory curvature