GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis
career value
181K/year
🤖 AI Summary
Existing evaluation methods struggle to effectively assess the dynamic execution capabilities of large language models (LLMs) in complex, multi-step geospatial analysis tasks and lack support for runtime feedback and the multimodal nature of spatial outputs. To address this, this work introduces a dynamic, interactive benchmark tailored for tool-augmented GIS agents, encompassing 117 atomic GIS tools and 53 representative tasks. It proposes the Parameter Execution Accuracy (PEA) metric, a “Last-Try Alignment” strategy, and a vision-language model–based mechanism for validating spatial outputs. Furthermore, the study designs a Plan-and-React agent architecture that decouples global planning from local reactive execution. Experimental results demonstrate that this architecture significantly outperforms baseline approaches across seven mainstream LLMs, achieving both logical rigor in multi-step reasoning and robustness in error recovery.
Application Category
📝 Abstract
The integration of Large Language Models (LLMs) into Geographic Information Systems (GIS) marks a paradigm shift toward autonomous spatial analysis. However, evaluating these LLM-based agents remains challenging due to the complex, multi-step nature of geospatial workflows. Existing benchmarks primarily rely on static text or code matching, neglecting dynamic runtime feedback and the multimodal nature of spatial outputs. To address this gap, we introduce GeoAgentBench (GABench), a dynamic and interactive evaluation benchmark tailored for tool-augmented GIS agents. GABench provides a realistic execution sandbox integrating 117 atomic GIS tools, encompassing 53 typical spatial analysis tasks across 6 core GIS domains. Recognizing that precise parameter configuration is the primary determinant of execution success in dynamic GIS environments, we designed the Parameter Execution Accuracy (PEA) metric, which utilizes a "Last-Attempt Alignment" strategy to quantify the fidelity of implicit parameter inference. Complementing this, a Vision-Language Model (VLM) based verification is proposed to assess data-spatial accuracy and cartographic style adherence. Furthermore, to address the frequent task failures caused by parameter misalignments and runtime anomalies, we developed a novel agent architecture, Plan-and-React, that mimics expert cognitive workflows by decoupling global orchestration from step-wise reactive execution. Extensive experiments with seven representative LLMs demonstrate that the Plan-and-React paradigm significantly outperforms traditional frameworks, achieving the optimal balance between logical rigor and execution robustness, particularly in multi-step reasoning and error recovery. Our findings highlight current capability boundaries and establish a robust standard for assessing and advancing the next generation of autonomous GeoAI.
Problem
Research questions and friction points this paper is trying to address.
tool-augmented agents
spatial analysis
dynamic execution
parameter accuracy
geospatial workflows
Innovation
Methods, ideas, or system contributions that make the work stand out.
GeoAgentBench
Parameter Execution Accuracy
Plan-and-React
tool-augmented agents
Vision-Language Model
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
