AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

2026-05-25

AI Summary

This work addresses the insufficient robustness of existing multimodal large language model–based computer-use agents in real-world dynamic environments, where non-adversarial perturbations—such as pop-up windows or resolution changes—can significantly disrupt task execution. To this end, we formally define and quantify agent vulnerability under such environmental disturbances for the first time, introducing AgentHijack, a benchmark comprising nine configurable interference types that simulate execution interruptions. We further propose AgentHijack-Agent, a dual-module framework integrating environment monitoring and action generation to enhance perceptual grounding and behavioral adaptability. Experiments demonstrate that even minor perturbations drastically degrade the performance of current agents, whereas our approach substantially improves robustness across diverse desktop tasks. Code and data are publicly released.

Abstract

Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups, resolution changes, and competing applications frequently interfere with agent perception and control. We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions, where the uncertainties in dynamic environment disrupt the execution flow without direct adversarial intent. Specifically, AgentHijack introduces 9 configurable common corruptions to replicate realistic imperfect scenarios. We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation, which emphasizes the fragility of agents and underscores the necessity of robustness evaluation. Afterward, we propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking. Extensive experiments validate its effectiveness. Our code, environment, baseline models and data are publicly available at: https://AgentHijack.github.io.

Problem

• computer-use agents
• robustness
• environment corruptions
• multimodal large language models
• benchmarking

Innovation

• AgentHijack
• robustness benchmark
• multimodal LLM agents
• environment corruption
• computer use automation