Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing

2026-03-06

AI Summary

This work proposes the first multimodal agent framework tailored for spreadsheet understanding, addressing the limitations of existing methods that struggle with multi-step reasoning due to context loss, compression artifacts, or exceeding model context windows when processing complex enterprise spreadsheets. The framework employs a planner-driven iterative tool-calling mechanism, integrating multimodal retrieval-augmented generation (RAG) with the NVIDIA NeMo Retriever 1B embedding model to enable end-to-end operations—from complex analytical reasoning to structured editing—without information loss. Evaluated on three benchmarks—FRTR-Bench, SpreadsheetLLM, and FINCH—the approach outperforms prior state-of-the-art methods by 25, 7, and 32 percentage points, respectively. Expert evaluations spanning over 200 hours further confirm significant improvements in both reasoning capability and auditability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advances in multimodal Retrieval-Augmented Generation (RAG) enable Large Language Models (LLMs) to analyze enterprise spreadsheet workbooks containing millions of cells, cross-sheet dependencies, and embedded visual artifacts. However, state-of-the-art approaches exclude critical context through single-pass retrieval, lose data resolution through compression, and exceed LLM context windows through naive full-context injection, preventing reliable multi-step reasoning over complex enterprise workbooks. We introduce Beyond Rows to Reasoning (BRTR), a multimodal agentic framework for spreadsheet understanding that replaces single-pass retrieval with an iterative tool-calling loop, supporting end-to-end Excel workflows from complex analysis to structured editing. Supported by over 200 hours of expert human evaluation, BRTR achieves state-of-the-art performance across three frontier spreadsheet understanding benchmarks, surpassing prior methods by 25 percentage points on FRTR-Bench, 7 points on SpreadsheetLLM, and 32 points on FINCH. We evaluate five multimodal embedding models, identifying NVIDIA NeMo Retriever 1B as the top performer for mixed tabular and visual data, and vary nine LLMs. Ablation experiments confirm that the planner, retrieval, and iterative reasoning each contribute substantially, and cost analysis shows GPT-5.2 achieves the best efficiency-accuracy trade-off. Throughout all evaluations, BRTR maintains full auditability through explicit tool-call traces.

Problem

• multimodal RAG
• spreadsheet understanding
• multi-step reasoning
• context window limitation
• data resolution loss

Innovation

• Agentic Retrieval
• Multimodal RAG
• Iterative Reasoning
• Spreadsheet Understanding
• Tool-calling Framework