Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation

2026-05-28

AI Summary

Existing approaches struggle to evaluate large language models’ generated interactive webpages efficiently and holistically, as they rely either on manual assessment or rigid metrics that fail to capture human-like holistic judgments of dynamic user experiences. This work proposes a reference-free, autonomous holistic evaluation paradigm: we introduce Cookie-Bench, a WebDev benchmark comprising 1,000 prompt-robust queries spanning 11 domains, and design a three-stage framework grounded in Flavell’s metacognitive theory. The framework integrates static perception, agent-driven interaction, and multimodal evidence—including videos, audio, and screenshots—to deliver dual-dimensional scores for both aesthetics and functionality, along with structured attribution. Experiments demonstrate strong alignment with expert ratings and reveal substantial room for improvement among 13 state-of-the-art models in generating interactive web content.

Abstract

Front-end web code has become a core product surface for every frontier LLM release, yet evaluating these interactive applications at development speed remains costly because human-judged leaderboards like Arena do not scale. Existing automated proxies typically lean on reference implementations, test suites, or rigid checklists, and tend to miss the reasoned synthesis a human reviewer performs over a live session. We articulate a new evaluation regime that is simultaneously reference-free, autonomously driven, and holistically reasoned, and instantiate it through two artifacts. \textbf{\dataname} is an 11-domain, 54-leaf, 1,000-query WebDev benchmark spanning both static-presentation and interactive-application tasks, balanced across three difficulty tiers and three target-language groups, with briefs rewritten to resist recall from circulated prompts. \textbf{\framename}, grounded in Flavell's metacognitive monitoring, separates evidence accumulation from judgment across three stages: Static Perception forms a first impression from passive observation; Agent-Driven Interaction explores the application autonomously while capturing continuous screen video, audio, and per-step screenshots; Dynamic Scoring issues holistic functionality and aesthetics verdicts with structured failure attribution only after the evidence chain is complete. On \dataname, \framename aligns closely with expert human ratings while surfacing substantial headroom across 13 frontier LLMs on interactive web generation. \noindenthttps://anonymous.4open.science/r/Cookie-3CE/

Problem

• web generation
• interactive evaluation
• LLM benchmarking
• reference-free assessment
• front-end code

Innovation

• reference-free evaluation
• agent-driven interaction
• metacognitive monitoring
• holistic reasoning
• interactive web generation