SheetDesigner: MLLM-Powered Spreadsheet Layout Generation with Rule-Based and Vision-Based Reflection

2025-09-09

AI Summary

Existing automated layout models fail to accommodate spreadsheet-specific constraints: their continuous coordinate modeling disregards the inherent discrete grid structure, and they neglect semantic relationships such as data dependencies and contextual links. Method: This paper formally defines the spreadsheet layout generation task for the first time and proposes a zero-shot, training-free approach leveraging multimodal large language models (MLLMs). It integrates rule-based reflection—ensuring alignment and non-overlap—with visual reflection—optimizing balance and readability—to overcome limitations of pure rectangular, continuous-space modeling. Contribution/Results: We introduce a seven-criterion evaluation protocol and validate our method on a dataset of 3,326 real-world spreadsheets. Compared to five baseline methods, our approach achieves ≥22.6% improvement in layout quality, generating structurally sound and semantically coherent complex layouts. Code and data are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Spreadsheets are critical to data-centric tasks, with rich, structured layouts that enable efficient information transmission. Given the time and expertise required for manual spreadsheet layout design, there is an urgent need for automated solutions. However, existing automated layout models are ill-suited to spreadsheets, as they often (1) treat components as axis-aligned rectangles with continuous coordinates, overlooking the inherently discrete, grid-based structure of spreadsheets; and (2) neglect interrelated semantics, such as data dependencies and contextual links, unique to spreadsheets. In this paper, we first formalize the spreadsheet layout generation task, supported by a seven-criterion evaluation protocol and a dataset of 3,326 spreadsheets. We then introduce SheetDesigner, a zero-shot and training-free framework using Multimodal Large Language Models (MLLMs) that combines rule and vision reflection for component placement and content population. SheetDesigner outperforms five baselines by at least 22.6%. We further find that through vision modality, MLLMs handle overlap and balance well but struggle with alignment, necessitates hybrid rule and visual reflection strategies. Our codes and data is available at Github.

Problem

• Automating spreadsheet layout generation to save time and expertise
• Addressing discrete grid structure and component semantics in spreadsheets
• Overcoming limitations of existing axis-aligned rectangle models

Innovation

• MLLM-powered zero-shot framework
• Rule and vision reflection hybrid
• Training-free spreadsheet layout generation