DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation

2026-08-11

AI Summary

This work addresses the lack of effective evaluation mechanisms for large language models (LLMs) in generating interactive analytical dashboards, which hinders the assessment of their analytical reasoning and interaction quality. The authors propose the first open-ended benchmark for interactive dashboard generation, requiring models to produce both a functional dashboard and a replayable interaction trace. Evaluation is automated through a browser-based executor and a vision-language model (VLM), with the interaction trace serving as a novel basis for judgment. A lightweight, open-source judge model, DashJudge-8B, combined with Bradley-Terry aggregation, enables scalable leaderboard construction. Experiments demonstrate that DashJudge-8B reliably replicates human judgments, that interaction evidence substantially improves evaluation consistency, and that state-of-the-art models still exhibit significant shortcomings in rendering fidelity, analytical depth, and interactive behavior.

Abstract

Analytic dashboards combine coordinated views and interactions for data exploration and decision-making. Recent models can generate them from data and natural-language goals, but evaluating their usefulness remains difficult. Dashboard generation is open-ended, and neither static appearance nor successful execution alone captures analytical support and interaction quality. We introduce DashArena, to our knowledge the first benchmark for open-ended, task-grounded generation of interactive analytic dashboards. Its key innovation is to require each system to generate both a dashboard and a replayable interaction trajectory. A browser executor replays the trajectory and turns the system's intended analytical workflow into reproducible visual and execution evidence. A VLM judge compares candidates using this evidence, and Bradley--Terry aggregation produces the leaderboard. We further distill the judge into the open-weight DashJudge-8B. Human evaluations show that DashJudge-8B effectively reproduces human judgments and ablations show that interaction evidence improves judge agreement. Experiments with frontier models reveal persistent rendering, analytical, and interaction failures. Together, these results show that realistic dashboard generation remains challenging and that interaction-aware evaluation captures failures missed by static or execution-only checks.

Problem

• interactive analytic dashboard
• LLM evaluation
• open-ended generation
• interaction quality
• benchmarking

Innovation

• interactive dashboard generation
• replayable interaction trajectory
• VLM-based evaluation
• DashJudge-8B
• task-grounded benchmarking