GridRoute: A Benchmark for LLM-Based Route Planning with Cardinal Movement in Grid Environments

2025-05-30

AI Summary

Large language models (LLMs) often overlook synergies with classical algorithms in pathfinding, leading to suboptimal reasoning and poor adherence to geometric constraints. Method: We propose Algorithm-Guided Reasoning—a novel paradigm integrating explicit algorithmic logic into LLM inference. We introduce GridRoute, the first grid-based benchmark for LLM path planning, supporting joint evaluation of correctness, optimality, and efficiency under 4-directional movement constraints. We further design Algorithm-of-Thought (AoT), a hybrid prompting technique that explicitly injects BFS/A* search logic into LLM reasoning chains. Our scalable evaluation framework unifies LLMs (7B–72B), classical algorithms, and structured prompts. Results: AoT boosts LLM pathfinding accuracy by 32.7% on average across complex grids and achieves 89.4% optimal solution rate—significantly outperforming Chain-of-Thought (CoT) and Tree-of-Thought (ToT).

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in Large Language Models (LLMs) have demonstrated their potential in planning and reasoning tasks, offering a flexible alternative to classical pathfinding algorithms. However, most existing studies focus on LLMs' independent reasoning capabilities and overlook the potential synergy between LLMs and traditional algorithms. To fill this gap, we propose a comprehensive evaluation benchmark GridRoute to assess how LLMs can take advantage of traditional algorithms. We also propose a novel hybrid prompting technique called Algorithm of Thought (AoT), which introduces traditional algorithms' guidance into prompting. Our benchmark evaluates six LLMs ranging from 7B to 72B parameters across various map sizes, assessing their performance in correctness, optimality, and efficiency in grid environments with varying sizes. Our results show that AoT significantly boosts performance across all model sizes, particularly in larger or more complex environments, suggesting a promising approach to addressing path planning challenges. Our code is open-sourced at https://github.com/LinChance/GridRoute.

Problem

• Evaluating LLMs' synergy with traditional route planning algorithms
• Assessing LLM performance in grid-based pathfinding tasks
• Developing hybrid prompting for improved path planning solutions

Innovation

• Hybrid prompting technique AoT integrates traditional algorithms
• Benchmark GridRoute evaluates LLMs in grid environments
• AoT boosts performance in complex path planning