CLIP the Landscape: Automated Tagging of Crowdsourced Landscape Images

2025-06-13

AI Summary

To address the challenge of predicting fine-grained geographic context labels in sparsely populated regions of the UK—where point-of-interest (POI) data is scarce and street-level imagery is unavailable—this paper proposes a multimodal multi-label classification framework tailored for Geograph crowdsourced landscape images. The method innovatively fuses CLIP-based visual embeddings, latitude-longitude geocoding features, and image caption text representations within a lightweight classification head, enabling full-label matching across 49 granular geographic context categories. Experimental results demonstrate substantial improvements over unimodal baselines, with significant accuracy gains under Kaggle’s rigorous evaluation protocol. Furthermore, we publicly release an efficient single-machine fine-tuning pipeline, optimized for low-resource deployment. This work advances GeoAI by strengthening location-aware representation learning and facilitating spatial semantic understanding in data-scarce regions.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present a CLIP-based, multi-modal, multi-label classifier for predicting geographical context tags from landscape photos in the Geograph dataset--a crowdsourced image archive spanning the British Isles, including remote regions lacking POIs and street-level imagery. Our approach addresses a Kaggle competitionfootnote{https://www.kaggle.com/competitions/predict-geographic-context-from-landscape-photos} task based on a subset of Geograph's 8M images, with strict evaluation: exact match accuracy is required across 49 possible tags. We show that combining location and title embeddings with image features improves accuracy over using image embeddings alone. We release a lightweight pipelinefootnote{https://github.com/SpaceTimeLab/ClipTheLandscape} that trains on a modest laptop, using pre-trained CLIP image and text embeddings and a simple classification head. Predicted tags can support downstream tasks such as building location embedders for GeoAI applications, enriching spatial understanding in data-sparse regions.

Problem

• Predicting geographical tags from landscape photos
• Improving accuracy with multi-modal embeddings
• Supporting GeoAI applications in data-sparse regions

Innovation

• CLIP-based multi-modal multi-label classifier
• Combines location, title, and image embeddings
• Lightweight pipeline for modest hardware