Detecting Legend Items on Historical Maps Using GPT-4o with In-Context Learning

2025-10-09

AI Summary

Historical map legends exhibit highly variable layouts and unstructured formats, posing significant challenges for automatic symbol–text alignment. To address this, we propose the first layout-aware, end-to-end legend parsing framework. It first employs LayoutLMv3 for fine-grained layout detection, then introduces GPT-4o with in-context learning—leveraging exemplar-driven, structured JSON prompts to jointly predict symbol bounding boxes and align them precisely with corresponding textual descriptions. This work pioneers the application of multimodal large language models to structured parsing of historical map legends, substantially enhancing symbol–text matching accuracy. Evaluated on a standard benchmark, our method achieves 88% F1 score and 85% IoU—outperforming state-of-the-art OCR and segmentation baselines. Results validate the effectiveness of integrating prompt engineering, example-based guidance, and layout-aware multimodal modeling for historical cartographic analysis.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Historical map legends are critical for interpreting cartographic symbols. However, their inconsistent layouts and unstructured formats make automatic extraction challenging. Prior work focuses primarily on segmentation or general optical character recognition (OCR), with few methods effectively matching legend symbols to their corresponding descriptions in a structured manner. We present a method that combines LayoutLMv3 for layout detection with GPT-4o using in-context learning to detect and link legend items and their descriptions via bounding box predictions. Our experiments show that GPT-4 with structured JSON prompts outperforms the baseline, achieving 88% F-1 and 85% IoU, and reveal how prompt design, example counts, and layout alignment affect performance. This approach supports scalable, layout-aware legend parsing and improves the indexing and searchability of historical maps across various visual styles.

Problem

• Automating detection of legend items on historical maps with inconsistent layouts
• Matching legend symbols to descriptions in structured format using AI
• Improving indexing and searchability of historical maps across visual styles

Innovation

• Combines LayoutLMv3 for layout detection
• Uses GPT-4o with in-context learning
• Links legend items via bounding box predictions