CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation

2025-01-28

AI Summary

Current web navigation agents exhibit limitations in executing complex tasks and interpreting user intent, hindering efficient human-agent collaboration. This paper proposes a novel web navigation framework supporting both autonomous operation and real-time human-agent collaboration. It introduces the first dynamic control handover mechanism, enabling seamless transitions among human intervention, action override, and agent recovery. We design a collaborative evaluation paradigm that jointly optimizes task success rate and human participation level. The framework adopts a hybrid control architecture integrating DOM understanding, action space modeling, and interactive state management. Empirical evaluation across five major websites demonstrates a 95% task success rate, with humans performing only 15.2% of all actions; the agent independently contributes nearly 50% of successful execution paths. These results significantly enhance collaborative efficiency and interpretability.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Coordination and Collaboration
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Sustainability of Web economics
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent's capabilities effectively. We propose CowPilot, a framework supporting autonomous as well as human-agent collaborative web navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform only 15.2% of the total steps. Even with human interventions during task execution, the agent successfully drives up to half of task success on its own. CowPilot can serve as a useful tool for data collection and agent evaluation across websites, which we believe will enable research in how users and agents can work together. Video demonstrations are available at https://oaishi.github.io/cowpilot.html

Problem

• Human-Agent Collaboration
• Complex Task Handling
• User Intent Understanding

Innovation

• Collaborative Browsing
• Interruption Management
• Performance Evaluation