Structured Distillation of Web Agent Capabilities Enables Generalization

2026-04-09

AI Summary

While existing large language models can operate complex websites, their high computational cost and reliance on proprietary APIs hinder local deployment. This work proposes the Agent-as-Annotators framework, which for the first time systematically integrates human annotation workflows into agent trajectory synthesis. By structuring a modular teacher model—comprising task designers, annotators, and supervisors—the method generates high-quality reasoning trajectories. Coupled with quality filtering and evaluation prompts, it enables supervised fine-tuning of lightweight student models using only a single teacher model. Evaluated on WebArena, the approach achieves a 41.5% success rate, outperforming both Claude 3.5 Sonnet and GPT-4o, and demonstrates a remarkable 18.2-percentage-point improvement on the previously unseen enterprise platform WorkArena L1, setting a new state-of-the-art performance for open-source models.

Abstract

Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io

Problem

• web agents
• local deployment
• generalization
• cost efficiency
• third-party API dependency

Innovation

• structured distillation
• web agent
• trajectory synthesis
• modular LLM components
• local deployment