VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets

2026-09-10

AI Summary

本文通过引入VRL-Bench评估不同方法在有限尝试次数下的计算机控制任务表现，并提出VEX²调度器以平衡探索与利用，提高成功率。

Abstract

Learning from trial and error is a promising way to improve language agents on complex tasks such as computer control. Reflexion introduced verbal reinforcement learning, which turns failed trials into text that guides later attempts without updating model parameters. We introduce VRL-Bench, a harness for fair evaluation of trial-and-error learning under finite trial budgets. Across three models on MiniWoB and WebShop, we evaluate updates from several prominent verbal-memory methods spanning Reflexion and later work: each improves observed success over memory-free retry in some settings but reduces it in others. Replay experiments show that using reflection can reduce success rates, revealing a trade-off between exploiting experience and continued exploration. We propose VEX$^2$, a verbal exploration--exploitation scheduler that uses a language model to jointly select policies and allocate the remaining trial budget. VEX$^2$ is the only evaluated update to achieve positive observed success-rate gains over retry in all six settings.

Problem

• trial and error learning
• finite trial budgets
• verbal reinforcement learning
• success rate

Innovation

• VRL-Bench
• trial-and-error learning
• verbal reinforcement learning
• VEX²
• exploration-exploitation