Towards Trustworthy GUI Agents: A Survey

2025-03-30

AI Summary

This paper identifies five critical trustworthiness challenges confronting GUI agents: security vulnerabilities (e.g., susceptibility to adversarial attacks), reliability degradation in dynamic environments, lack of transparency and interpretability, ethical and regulatory non-compliance, and insufficient scientific rigor in evaluation—particularly the absence of realistic, scenario-based benchmarks and cascading failures in sequential decision-making. To address these, we propose a novel five-dimensional trustworthiness framework for GUI agents, moving beyond task accuracy toward a holistic, multi-faceted evaluation paradigm. Integrating principles from trustworthy AI, human–computer interaction, and robustness modeling, we construct a systematic challenge taxonomy and a principled mitigation roadmap. Our work provides both theoretical foundations and actionable guidelines for developing safety standards, establishing rigorous evaluation benchmarks, and enabling responsible deployment of GUI agents. (149 words)

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Adversarial Agents
• Humans and AI: Intelligent User Interfaces

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

GUI agents, powered by large foundation models, can interact with digital interfaces, enabling various applications in web automation, mobile navigation, and software testing. However, their increasing autonomy has raised critical concerns about their security, privacy, and safety. This survey examines the trustworthiness of GUI agents in five critical dimensions: security vulnerabilities, reliability in dynamic environments, transparency and explainability, ethical considerations, and evaluation methodologies. We also identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making, and a lack of realistic evaluation benchmarks. These issues not only hinder real-world deployment but also call for comprehensive mitigation strategies beyond task success. As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential. This survey provides a foundation for advancing trustworthy GUI agents through systematic understanding and future research.

Problem

• Examining GUI agents' trustworthiness in security, reliability, and ethics
• Addressing vulnerabilities to adversarial attacks and cascading failures
• Establishing safety standards for responsible GUI agent development

Innovation

• Survey examines GUI agent trustworthiness dimensions
• Identifies challenges like adversarial attack vulnerabilities
• Proposes robust safety standards for development