Democratizing planetary-scale analysis: An ultra-lightweight Earth embedding database for accurate and flexible global land monitoring

2026-01-16

AI Summary

This study addresses the challenge of high computational and storage costs that hinder global-scale Earth observation research. To overcome this, the authors propose an ultra-lightweight Earth Surface Embedding Database (ESD) spanning 2000–2024 at 30-meter resolution. By integrating Landsat and MODIS Terra data through a novel ESDNet architecture and Finite Scalar Quantization (FSQ), the method compresses annual phenological cycles into 12-step latent representations. This approach achieves a compression ratio of approximately 340×, reducing the storage requirement for a single year of global land surface data to just 2.4 TB while maintaining high reconstruction fidelity (MAE: 0.0130, RMSE: 0.0179, CC: 0.8543). Land cover classification accuracy reaches 79.74%, outperforming results from raw reflectance fusion, thereby enabling multi-decadal, few-shot, and cross-year consistent global analyses on standard workstations.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

The rapid evolution of satellite-borne Earth Observation (EO) systems has revolutionized terrestrial monitoring, yielding petabyte-scale archives. However, the immense computational and storage requirements for global-scale analysis often preclude widespread use, hindering planetary-scale studies. To address these barriers, we present Embedded Seamless Data (ESD), an ultra-lightweight, 30-m global Earth embedding database spanning the 25-year period from 2000 to 2024. By transforming high-dimensional, multi-sensor observations from the Landsat series (5, 7, 8, and 9) and MODIS Terra into information-dense, quantized latent vectors, ESD distills essential geophysical and semantic features into a unified latent space. Utilizing the ESDNet architecture and Finite Scalar Quantization (FSQ), the dataset achieves a transformative ~340-fold reduction in data volume compared to raw archives. This compression allows the entire global land surface for a single year to be encapsulated within approximately 2.4 TB, enabling decadal-scale global analysis on standard local workstations. Rigorous validation demonstrates high reconstructive fidelity (MAE: 0.0130; RMSE: 0.0179; CC: 0.8543). By condensing the annual phenological cycle into 12 temporal steps, the embeddings provide inherent denoising and a semantically organized space that outperforms raw reflectance in land-cover classification, achieving 79.74% accuracy (vs. 76.92% for raw fusion). With robust few-shot learning capabilities and longitudinal consistency, ESD provides a versatile foundation for democratizing planetary-scale research and advancing next-generation geospatial artificial intelligence.

Problem

• Earth Observation
• global land monitoring
• computational barriers
• data volume
• planetary-scale analysis

Innovation

• Earth embedding
• ultra-lightweight database
• Finite Scalar Quantization
• planetary-scale monitoring
• geospatial AI