UXBench: Measuring the Actionability of LLM-Generated UX Critiques

2026-06-15

AI Summary

This work addresses the current lack of controllable benchmarks for evaluating the reliability and actionability of user experience (UX) critiques generated by large language models, particularly across diverse interface scenarios. The authors propose UXBench, the first benchmark that assesses critique quality through downstream repair efficacy. It comprises ten categories of locally executable web components and incorporates a guided browser exploration mechanism, requiring models to produce structured UX reports grounded in interaction evidence. Report quality is measured by whether downstream repair agents can effectively improve interfaces based on these reports. UXBench introduces interaction-evidence constraints, a multidimensional scoring scheme, and blind human validation. Experiments across eight state-of-the-art models reveal that UX evaluation capability remains significantly multidimensional and unsaturated, with notable disparities in actionability, repair effectiveness, component reliability, and adaptability across interface types.

Abstract

Large language models (LLMs) are increasingly deployed as UX judges that inspect interfaces, diagnose usability problems, and propose repairs. Yet no controlled benchmark measures whether the resulting critiques are reliable and actionable across heterogeneous product surfaces. We introduce UXBench, a benchmark for evaluating LLMs as interaction-grounded UX judges. UXBench comprises local-first runnable web fixtures spanning ten product-surface families, paired with coverage-gated browser exploration that forces models to collect interaction evidence before reporting. Each judge model produces a structured UX report over seven rubric dimensions; report quality is measured by whether a fixed downstream repair agent can improve the interface based on the critique. We evaluate eight frontier models under both an automated repair-lift protocol and a blind human validation study. Results show that UX judging is neither saturated nor one dimensional: models differ meaningfully in report actionability, exhibit distinct rubric-level repair signatures, vary in fixture-level reliability, and trade leadership across surface categories

Problem

• UX critique
• actionability
• large language models
• benchmark
• usability evaluation

Innovation

• UXBench
• actionability
• LLM evaluation
• interaction-grounded critique
• repair-lift protocol