MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction

2026-01-19

AI Summary

This work addresses the vulnerability of Computer-Using Agents (CUAs) to malicious instruction or visual prompt injection attacks, which can trigger unsafe reasoning and harmful actions. Existing defenses often compromise utility by prematurely terminating tasks. To overcome this limitation, the authors propose MirrorGuard—a plug-and-play defense framework that introduces, for the first time, a neural-symbolic hybrid pipeline for pure-text GUI simulation. This approach efficiently generates high-risk interaction trajectories without requiring real-system execution, enabling agents to dynamically intercept and correct unsafe reasoning chains during actual deployment. Evaluated on ByteDance’s UI-TARS system, MirrorGuard reduces the unsafe operation rate from 66.5% to 13.0%, substantially outperforming the state-of-the-art GuardAgent (53.9%) while achieving a lower false rejection rate, thereby effectively balancing safety and task completion capability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Large foundation models are integrated into Computer Use Agents (CUAs), enabling autonomous interaction with operating systems through graphical user interfaces (GUIs) to perform complex tasks. This autonomy introduces serious security risks: malicious instructions or visual prompt injections can trigger unsafe reasoning and cause harmful system-level actions. Existing defenses, such as detection-based blocking, prevent damage but often abort tasks prematurely, reducing agent utility. In this paper, we present MirrorGuard, a plug-and-play defense framework that uses simulation-based training to improve CUA security in the real world. To reduce the cost of large-scale training in operating systems, we propose a novel neural-symbolic simulation pipeline, which generates realistic, high-risk GUI interaction trajectories entirely in a text-based simulated environment, which captures unsafe reasoning patterns and potential system hazards without executing real operations. In the simulation environment, MirrorGuard learns to intercept and rectify insecure reasoning chains of CUAs before they produce and execute unsafe actions. In real-world testing, extensive evaluations across diverse benchmarks and CUA architectures show that MirrorGuard significantly mitigates security risks. For instance, on the ByteDance UI-TARS system, it reduces the unsafe rate from 66.5% to 13.0% while maintaining a marginal false refusal rate (FRR). In contrast, the state-of-the-art GuardAgent only achieves a reduction to 53.9% and suffers from a 15.4% higher FRR. Our work proves that simulation-derived defenses can provide robust, real-world protection while maintaining the fundamental utility of the agent. Our code and model are publicly available at https://bmz-q-q.github.io/MirrorGuard/.

Problem

• Computer Use Agents
• Security Risks
• Visual Prompt Injection
• Unsafe Reasoning
• Autonomous GUI Interaction

Innovation

• simulation-to-real reasoning
• neural-symbolic simulation
• Computer Use Agents
• secure reasoning correction
• GUI interaction safety