CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception

2026-08-08

AI Summary

This work addresses a critical gap in existing browser agent evaluations, which commonly overlook the challenges of complex user interface interactions and visual perception of dynamic content in cross-site tasks. To bridge this gap, the authors propose the first cross-site benchmark that integrates demanding UI manipulation with visual understanding requirements. The framework abstracts real websites into structured “site cards” and synthesizes 420 multi-step workflow tasks spanning 24 domains and 108 websites through a decomposition–recomposition mechanism. It enables fine-grained capability diagnosis and incorporates verifiable agents as automated evaluators. Experimental results reveal that state-of-the-art agents achieve significantly low success rates on this benchmark, underscoring that perception-intensive interaction remains a fundamental bottleneck in web automation.

Abstract

Large language models are increasingly deployed as autonomous agents that interact with the web through browsers. While recent progress has been driven by benchmarks that evaluate end-to-end task success, these evaluations largely overlook two fundamental sources of difficulty in real web browsing: complex actions over rich user interfaces and visual perception of dynamically rendered content, especially in workflows that span multiple websites. We introduce CAP, a scalable benchmark for evaluating browser agents on cross-site, human-like web tasks that require non-trivial UI interactions and visual understanding. Specifically, we adopt a decomposition-and-recomposition pipeline that first abstracts each website into a structured site card capturing user-facing functions, complex execution operations, and perceptual requirements, and then recomposes these components into realistic cross-site workflows. Each task is therefore grounded in multiple specific operations on each website, enabling fine-grained diagnosis. Built on this framework, we construct 420 tasks across 108 real-world websites and 24 domains under careful quality control. Experiments on state-of-the-art browser agents using our verifiable agent-as-a-judge evaluation framework show low success rates and reveal that perception-heavy interactions remain a major bottleneck, exposing substantial gaps between current agents and real-world web browsing demands.

Problem

• browser agents
• complex actions
• visual perception
• cross-site tasks
• web interaction

Innovation

• cross-site browser agents
• complex UI interactions
• visual perception
• decomposition-and-recomposition
• scalable benchmark