EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents

2026-05-02

AI Summary

This work addresses the limitations of existing Earth observation (EO) benchmarks, which are typically confined to single-round, fixed-input tasks and fail to capture the dynamic, multimodal, temporal, and cross-sensor decision-making processes inherent in real-world EO analysis. To bridge this gap, we propose EO-Gym—a Gymnasium-style executable interactive environment that formulates EO as a planning problem involving evidence collection across geospatial, temporal, and sensor modalities. We introduce EO-Gym-Data, a large-scale benchmark comprising 9,078 expert trajectories, and present the first framework supporting multimodal tool invocation, encompassing six task families and 35 EO-specific tools. The framework integrates multimodal file indexing, an executable tool library, fine-tuning of vision-language models (e.g., Qwen3-VL-4B-Instruct), and trajectory-level evaluation protocols. Experiments show that our fine-tuned EO-Gym-4B model improves Pass@3 from 0.49 to 0.74 under the primary evaluation setting, while also revealing significant shortcomings of general-purpose vision-language models in temporal and cross-modal EO reasoning.

Abstract

Earth Observation (EO) analysis is inherently interactive: resolving uncertainty often requires expanding the region of interest, retrieving historical observations, and switching across sensors such as optical and Synthetic Aperture Radar. However, most EO benchmarks collapse this process into fixed-input, single-turn tasks. To address this gap, we present EO-Gym, a controlled executable framework for multimodal, tool-using EO agents that formulates EO analysis as a Gymnasium-style local geospatial workspace backed by more than 660k multimodal files indexed by location, time, and sensor type, with 35 EO-specialized tools spanning six task families. Built on this environment, we construct EO-Gym-Data, a benchmark of 9,078 trajectories and 34,604 reasoning steps, and grounded in eight public EO datasets together with Landsat and Sentinel-2 imagery. Evaluating $10$ open and closed VLMs shows that strong general-purpose models still struggle with interactive EO reasoning, especially on temporal and cross-modal workflows. As a reference baseline, EO-Gym-4B, obtained by fine-tuning Qwen3-VL-4B-Instruct on EO-Gym-Data, improves overall Pass@3 from $0.49$ to $0.74$ under the main evaluation setting. O-Gym provides a reproducible environment for interactive EO agents, operationalizing EO as an evidence-gathering problem that requires planning across geospatial, temporal, and sensing modality.

Problem

• Earth Observation
• interactive reasoning
• multimodal analysis
• benchmarking
• geospatial reasoning

Innovation

• Earth Observation
• Multimodal Agents
• Interactive Reasoning
• Geospatial Tools
• Vision-Language Models