WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games

2026-05-17

AI Summary

This work addresses a critical gap in the evaluation of code-generating agents, which has predominantly focused on source code or intermediate artifacts rather than end-to-end validation of fully deliverable applications. The authors propose the first requirement-to-application benchmarking framework based on native browser games, automatically transforming structured game specifications into executable applications and evaluating them interactively within real browsers via a standardized deployment protocol. Evaluation relies on a three-tier runtime labeling scheme—EXCELLENT, USABLE, and UNUSABLE—that demonstrates high alignment with human judgments. Benchmarking 12 coding agents across 111 tasks reveals that while the best-performing configuration achieves a 76.9% usability rate, only 20.2% of generated applications meet the EXCELLENT criterion, underscoring a substantial gap in current systems’ ability to fully satisfy user requirements.

Abstract

Coding agents are increasingly used as application builders, yet many evaluations still focus on source code, repository-level tests, or intermediate traces rather than the delivered application. We introduce WebGameBench, a requirement-to-application benchmark that evaluates whether coding agents can turn a frozen Structured WebGame Specification into a browser-accessible game. Browser-native games provide a compact but behavior-dense testbed: even simple games require coordinated input handling, spatial mapping, rule execution, state transitions, terminal conditions, restart behavior, and visible feedback. In WebGameBench, each generated artifact is built, served, and exposed as a browser-accessible application under a unified deployment protocol. A runtime evaluator then interacts with the delivered game in a real browser and assigns a three-way label: EXCELLENT, USABLE, or UNUSABLE. On a human-reviewed subset, the runtime label is broadly aligned with human gameplay review under the Usable-rate criterion. Across 111 tasks, 12 coding agents, and 14 evaluation configurations, WebGameBench separates current systems: the best configuration reaches a 76.9% usable rate but only a 20.2% excellent rate. This gap shows that crossing the minimum playable-delivery threshold is still far from complete requirement satisfaction. To our knowledge, WebGameBench is the first requirement-to-application benchmark for browser-native game delivery that validates delivered-application runtime labels against independent human gameplay review under the Usable-rate criterion.

Problem

• coding agents
• application evaluation
• requirement-to-application
• browser-native games
• runtime validation

Innovation

• requirement-to-application evaluation
• coding agents
• browser-native games
• runtime evaluation
• structured specification