GPSBench: Do Large Language Models Understand GPS Coordinates?

2026-02-17

AI Summary

This study addresses the limited native understanding of GPS coordinates and geospatial relationships in large language models (LLMs) within real-world applications. To systematically evaluate such capabilities without external tools, the authors introduce GPSBench, a novel benchmark comprising 57,800 samples across 17 diverse tasks. Through zero-shot and fine-tuned evaluations, noise robustness tests, and downstream transfer experiments, they find that LLMs exhibit relatively strong country-level geolocation accuracy but weaker performance at the city level, alongside generally poor geometric reasoning abilities. While coordinate-aware input augmentation enhances downstream task performance, fine-tuning risks degrading the model’s pre-existing world knowledge. This work establishes a new foundation for assessing and advancing geospatial intelligence in language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) are increasingly deployed in applications that interact with the physical world, such as navigation, robotics, or mapping, making robust geospatial reasoning a critical capability. Despite that, LLMs' ability to reason about GPS coordinates and real-world geography remains underexplored. We introduce GPSBench, a dataset of 57,800 samples across 17 tasks for evaluating geospatial reasoning in LLMs, spanning geometric coordinate operations (e.g., distance and bearing computation) and reasoning that integrates coordinates with world knowledge. Focusing on intrinsic model capabilities rather than tool use, we evaluate 14 state-of-the-art LLMs and find that GPS reasoning remains challenging, with substantial variation across tasks: models are generally more reliable at real-world geographic reasoning than at geometric computations. Geographic knowledge degrades hierarchically, with strong country-level performance but weak city-level localization, while robustness to coordinate noise suggests genuine coordinate understanding rather than memorization. We further show that GPS-coordinate augmentation can improve in downstream geospatial tasks, and that finetuning induces trade-offs between gains in geometric computation and degradation in world knowledge. Our dataset and reproducible code are available at https://github.com/joey234/gpsbench

Problem

• GPS coordinates
• geospatial reasoning
• large language models
• geographic knowledge
• coordinate understanding

Innovation

• geospatial reasoning
• GPS coordinates
• large language models
• benchmark dataset
• coordinate understanding