HANSEL: Extracting Breadcrumbs from Web Agent Trajectories for Interactive Verification

2026-06-17

AI Summary

Current AI web agents rely on passive inspection of execution logs for multi-step task verification, which often leads to information overload and low interpretability. This work proposes an interactive verification approach that identifies critical evidence pages from agent trajectories and reconstructs lightweight snapshots preserving contextual states—such as filtering criteria, search queries, and scroll positions—to enable users to actively trace decision-making through a navigable visual interface, with explicit alerts when provenance is unavailable. The method achieves the first trajectory summarization technique that retains contextual fidelity while supporting verifiable traceability, attaining 83.7% precision and 88.8% recall on AssistantBench and Online-Mind2Web, along with a 61.6% reduction in trajectory size. User studies demonstrate significant reductions in verification time and cognitive load, alongside improved usability and error detection capability.

Abstract

AI web agents can perform complex, multi-step tasks such as searching for products, comparing options, and making purchases on behalf of users. However, verifying the correctness of an agent's output remains difficult. Existing transparency mechanisms, including full trajectory logs, source links, screenshots, and LLM-generated summaries, treat verification as a passive reading task, leaving users to sift through overwhelming logs or trust potentially unfaithful explanations. We present HANSEL (Highlighting Agent Navigation Steps as Evidence Links), a system that extracts interactive, verifiable evidence from web-agent trajectories. Given an agent trajectory, HANSEL extracts evidence pages and snippets and presents them as navigable, interactive views with relevant page state preserved (e.g., applied filters, search queries, and scroll positions), enabling users to verify how the agent arrived at its answer. When the agent's answer cannot be traced to any visited page, HANSEL explicitly flags this gap. A technical evaluation on 45 tasks from AssistantBench and Online-Mind2Web shows that HANSEL achieves 83.7% precision and 88.8% recall in identifying evidence pages, while reducing trajectory volume by 61.6%. In a controlled user study with 14 participants, HANSEL significantly reduced task completion time and perceived effort compared to a standard agent interface, while participants rated it significantly higher on usability, verification ease, and error identification. Our results demonstrate that reframing verification as an interactive activity, rather than passive consumption of agent explanations, leads to more efficient human oversight of AI agents.

Problem

• AI web agents
• verification
• transparency
• interactive verification
• agent trajectories

Innovation

• interactive verification
• web agent trajectories
• evidence extraction
• human-AI oversight
• trajectory summarization