SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks

2026-02-16

arXiv.org

AI Summary

This work addresses the challenges large language models (LLMs) face in end-to-end spreadsheet generation—specifically, their difficulty in simultaneously satisfying users’ explicit and implicit constraints and the absence of a unified evaluation benchmark. To this end, we introduce SpreadsheetArena, the first systematic evaluation framework grounded in human preferences. The platform employs blind pairwise preference voting augmented by expert assessments and multidimensional feature analysis to comprehensively evaluate generated workbooks across structural, interactive, and layout dimensions. Our study uncovers significant differences in style, functionality, and professionalism among outputs produced under varying prompting strategies. Furthermore, we release the first large-scale spreadsheet dataset encompassing prompts, model-generated outputs, and human preference annotations, thereby establishing spreadsheets as a compelling domain for investigating complex LLM capabilities.

Abstract

Large language models (LLMs) are increasingly tasked with producing and manipulating structured artifacts. We consider the task of end-to-end spreadsheet generation, where language models are prompted to produce spreadsheet artifacts to satisfy users'explicit and implicit constraints, specified in natural language. We introduce SpreadsheetArena, a platform for evaluating models'performance on the task via blind pairwise evaluations of LLM-generated spreadsheet workbooks. As with other complex, open-ended tasks, relevant evaluation criteria can vary substantially across use cases and prompts, often in ways that are difficult to formalize. Compared to general chat or text generation settings, spreadsheet generation presents unique challenges and opportunities: the task output structure is well-defined and multi-dimensional, and there are often complex considerations around interactivity and layout. Among other findings, we observe that stylistic, structural, and functional features of preferred spreadsheets vary substantially across use cases, and expert evaluations of spreadsheets for finance prompts suggests that even highly ranked arena models do not reliably produce spreadsheets aligned with domain-specific best practices. Our hope is that our work prompts further study of end-to-end spreadsheet generation as a challenging and interesting category of complex, open-ended tasks for LLMs. Our live arena is hosted at https://spreadsheetarena.ai.

Problem

• spreadsheet generation
• LLM evaluation
• preference modeling
• open-ended tasks
• domain-specific best practices

Innovation

• spreadsheet generation
• preference evaluation
• LLM benchmarking
• structured output generation
• domain-specific best practices