Problem
Research questions and friction points this paper is trying to address.
web agents
long-horizon tasks
hidden failures
completion-correctness gap
trace diagnostics
When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration
🤖 AI Summary
This work addresses the challenge that long-horizon web agents often produce incorrect outputs—manifesting as missing fields, unsupported claims, or reliance on outdated evidence—even when appearing to complete tasks successfully. To investigate this, the authors introduce Parallel WebBench, a benchmark comprising 1,679 verified task records, and train a WebExplorer-style agent using the GRPO algorithm with 16k context windows and up to 16 interactive turns per task through multi-path parallel exploration. Leveraging reproducible triggering mechanisms and trajectory-level diagnostics, they identify three persistent failure modes: context-limited search loops, premature termination on partial answers, and post-hoc evidence synthesis collapse. The best-performing model achieves a task completion rate of 96.0% (up from 50.7%) and an element-level F1 of 0.4529, yet its binary accuracy remains substantially lower than its completion rate, revealing a critical gap between task completion and factual correctness and underscoring the need for fine-grained evaluation grounded in evidence coverage and synthesis fidelity.
📝 Abstract
Long-horizon web agents often fail in ways hidden by final-answer evaluation: they may visit useful pages, produce a well-formed answer, and terminate confidently while still missing fields, over-including unsupported items, or relying on stale evidence. We study these failures with Parallel WebBench, a parallel web-exploration benchmark containing 1,679 verified records: 350 manually curated parallel tasks and 1,329 reconstructed records with verified URL-based trajectories. We train WebExplorer-style agents with GRPO under human-only, balanced human-synthetic, and synthetic-heavy data mixtures. At 16k context and 16 interaction rounds, the best GRPO model improves completion over WebExplorer-8B from 50.7% to 96.0% and GPT-4.1-mini-judged element-wise F1 from 0.2489 to 0.4529, but binary accuracy remains far below completion. Trace-level analysis identifies three persistent failure modes: context-bound search loops, premature termination on partial answers, and synthesis collapse after relevant evidence has already been retrieved. These results show that synthetic-data GRPO reduces abstention and improves partial correctness, but leaves a completion-correctness gap that requires evidence-grounded coverage and synthesis diagnostics.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Parallel Web Exploration
Trace Diagnostics
GRPO
Synthetic Data Augmentation
Failure Mode Analysis
🔎 Similar Papers
No similar papers found.
