A Multimodal Conversational Assistant for the Characterization of Agricultural Plots from Geospatial Open Data

2025-09-22

AI Summary

Non-expert users face significant challenges in efficiently leveraging open Earth observation (EO) and agricultural remote sensing data for sustainable land management. Method: We propose an open-source multimodal conversational assistant built upon a retrieval-augmented generation (RAG) framework that integrates orthoimagery, Sentinel-2 vegetation indices, and textual agricultural documentation. The system employs Qwen3-32B as its foundation language model to enable zero-shot natural language interaction. Crucially, we introduce an LLM-as-a-judge unsupervised, multi-dimensional automated evaluation mechanism to enhance reproducibility and cross-regional generalizability. Contribution/Results: Experiments demonstrate that the assistant generates accurate, context-aware, and interpretable responses characterizing individual farmland parcels. Comprehensive evaluation across multiple dimensions confirms the system’s effectiveness, robustness, and scalability. By abstracting technical complexity, it substantially lowers the barrier to entry for agricultural remote sensing data utilization among non-specialist stakeholders.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The increasing availability of open Earth Observation (EO) and agricultural datasets holds great potential for supporting sustainable land management. However, their high technical entry barrier limits accessibility for non-expert users. This study presents an open-source conversational assistant that integrates multimodal retrieval and large language models (LLMs) to enable natural language interaction with heterogeneous agricultural and geospatial data. The proposed architecture combines orthophotos, Sentinel-2 vegetation indices, and user-provided documents through retrieval-augmented generation (RAG), allowing the system to flexibly determine whether to rely on multimodal evidence, textual knowledge, or both in formulating an answer. To assess response quality, we adopt an LLM-as-a-judge methodology using Qwen3-32B in a zero-shot, unsupervised setting, applying direct scoring in a multi-dimensional quantitative evaluation framework. Preliminary results show that the system is capable of generating clear, relevant, and context-aware responses to agricultural queries, while remaining reproducible and scalable across geographic regions. The primary contributions of this work include an architecture for fusing multimodal EO and textual knowledge sources, a demonstration of lowering the barrier to access specialized agricultural information through natural language interaction, and an open and reproducible design.

Problem

• Lowering technical barriers for non-experts accessing agricultural geospatial data
• Enabling natural language interaction with multimodal agricultural datasets
• Integrating Earth Observation data with user documents through conversational AI

Innovation

• Multimodal retrieval and LLMs for natural language interaction
• RAG architecture combining orthophotos, indices and documents
• LLM-as-a-judge methodology for multi-dimensional evaluation