Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework

2026-05-28

AI Summary

This work addresses the high inference latency of large language model–driven web agents caused by verbose HTML observations and the lack of efficient evaluation methods for observation reduction techniques that balance latency and task performance. The authors propose a lightweight evaluation framework based on Minimal Failing Sets (MFS), introducing MFS retention rate as a novel proxy metric that eliminates the need for end-to-end reasoning or real webpage interactions, thereby accelerating evaluation by over two orders of magnitude. Experiments demonstrate that this approach achieves 2.2× and 3.1× single-step latency speedups on WorkArena L1 and WebLinx, respectively, while preserving 84% and 89% task success rates. The study further reveals that extractive reduction strategies require either substantial computational resources or domain-specific optimization to effectively reconcile efficiency with performance.

Abstract

HTML observations in LLM-based web agents are extremely long, and while many reduction methods have been proposed, it remains unclear which methods reduce overall agent latency while maintaining performance. The main obstacle is the high cost of end-to-end evaluation: in our experiments, evaluating 11 methods across 32 configurations on 33 tasks of WorkArena L1 required 232.4 cumulative hours. To address this, we propose a lightweight evaluation framework based on the Minimal Failure Set (MFS), the minimal set of HTML elements whose removal causes task failure. We define coverage as the fraction of instances in which a reduction method fully retains the MFS, which serves as a proxy metric that requires neither web access nor LLM inference. We validate that coverage strongly correlates with end-to-end success rate, with over 100$\times$ speedup in cumulative evaluation time on both benchmarks. Using this framework, we find that extractive HTML reduction methods require either high computation cost or domain-specific optimization to reduce agent latency while maintaining performance. Building on this, we optimize a pruning program on MFS training data, achieving 2.2$\times$ faster per-step latency on WorkArena L1 while retaining 84\% of the original success rate, and 3.1$\times$ faster on WebLinx while retaining 89\%.

Problem

• observation reduction
• web agents
• HTML processing
• latency
• LLM-based agents

Innovation

• Minimal Failure Set
• HTML reduction
• lightweight evaluation
• web agents
• coverage metric