Problem
Research questions and friction points this paper is trying to address.
image-triggered command injection
computer-use agents
visual patch
environmental consequences
multimodal
AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents
🤖 AI Summary
本文通过在网页中嵌入视觉补丁，测试其能否诱导计算机使用代理执行恶意命令，评估了五种开源或公开的GUI代理和视觉-语言模型后端。
📝 Abstract
This paper presents an end-to-end evaluation framework for image-triggered command injection against computer-use agents (CUAs). The goal is to test whether a local visual patch can induce verifiable environmental consequences along the full chain of screenshot input, VLM generation, action parsing, and environment execution. We train and deploy patches on author-controlled GitHub Pages pages and a locally deployed CSDN clone, and evaluate them in real environments across five open-source or publicly available GUI-agent or vision-language-model (VLM) backends. Our experiment aggregates 600 instance-level online cases, with T-ASR, TAPR, and E2E-ASR reaching 84.5%, 47.0%, and 20.3%, respectively. Trajectory analysis further shows that in some successful cases the agent first executes a malicious terminal command and then continues the original benign task. These results indicate that optimized local visual signals can affect not only VLM outputs but also propagate through the execution pipeline of open CUAs and create real environmental risk.
Innovation
Methods, ideas, or system contributions that make the work stand out.
image-triggered command injection
multimodal computer-use agents
visual patch attacks
end-to-end evaluation framework
environmental risk
🔎 Similar Papers
