Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsing

2025-05-15

AI Summary

This study investigates unintended consequences (UCs) arising from large language model (LLM)-driven GUI agents in human–computer collaborative web browsing. Drawing on 221 social media posts and 14 in-depth interviews, we propose a novel “phenomenon–impact–mitigation” analytical framework. We systematically identify five core failure phenomena—including instruction misinterpretation, dynamic interface interaction failures, and inadequate error handling—and trace their cascading four-tier impacts: task failure → user frustration → privacy/security risks → erosion of trust. Critically, we introduce user-centered mitigation strategies and distill three foundational design principles for GUI agents: robustness, human-centeredness, and explainability. These contributions advance both theoretical understanding and practical development of trustworthy, human–agent collaborative interfaces. (136 words)

Technology Category

• Humans and AI: Intelligent User Interfaces
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

The proliferation of Large Language Model (LLM)-based Graphical User Interface (GUI) agents in web browsing scenarios present complex unintended consequences (UCs). This paper characterizes three UCs from three perspectives: phenomena, influence and mitigation, drawing on social media analysis (N=221 posts) and semi-structured interviews (N=14). Key phenomenon for UCs include agents' deficiencies in comprehending instructions and planning tasks, challenges in executing accurate GUI interactions and adapting to dynamic interfaces, the generation of unreliable or misaligned outputs, and shortcomings in error handling and feedback processing. These phenomena manifest as influences from unanticipated actions and user frustration, to privacy violations and security vulnerabilities, and further to eroded trust and wider ethical concerns. Our analysis also identifies user-initiated mitigation, such as technical adjustments and manual oversight, and provides implications for designing future LLM-based GUI agents that are robust, user-centric, and transparent, fostering a crucial balance between automation and human oversight.

Problem

• Characterizing unintended consequences in human-GUI agent collaboration for web browsing
• Analyzing phenomena, influence, and mitigation of GUI agent deficiencies
• Addressing privacy, security, and trust issues in LLM-based GUI agents

Innovation

• Analyzing unintended consequences via social media and interviews
• Identifying GUI agent deficiencies in task comprehension
• Proposing robust, user-centric, transparent agent designs