Plover: Steering GUI Agents through Plan-Centric Interaction

2026-07-16

AI Summary

This work proposes a vision-based GUI automation system centered on explicit task planning to address the tendency of existing agents to deviate from user intent in dynamic interfaces and their lack of transparent, intervenable planning mechanisms. By treating task plans as persistent, inspectable, and editable external artifacts, the system adopts a planning-execution decoupled architecture that integrates multimodal visual inputs, screenshot-anchored interventions, and natural language guidance. This design enables real-time monitoring and localized correction of execution trajectories. Experimental results demonstrate that the approach effectively recovers from the majority of automation failures, substantially enhancing the system’s transparency, controllability, and adaptability in complex, evolving graphical user interfaces.

Abstract

Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.

Problem

• GUI automation
• user intent alignment
• plan visibility
• agent supervision
• interactive correction

Innovation

• plan-centric
• GUI automation
• replanning
• vision-based agents
• interactive correction