NeuCo-Bench: A Novel Benchmark Framework for Neural Embeddings in Earth Observation

2025-10-19

AI Summary

Neural compression and representation learning in Earth observation (EO) lack standardized evaluation protocols and are prone to pretraining bias. Method: We propose NeuCo-Bench—the first community-driven benchmark framework for EO neural embeddings. It introduces a task-agnostic, fixed-dimensional neural embedding interface to enable reusable evaluation pipelines; a “hidden-task” leaderboard mechanism that dynamically assesses embeddings without prior knowledge of downstream tasks, mitigating pretraining bias; and a balanced scoring system jointly optimizing accuracy and stability. Built upon the SSL4EO-S12-downstream multispectral, multitemporal dataset, NeuCo-Bench supports self-supervised foundation models. Results: Validated in the CVPR 2025 EARTHVISION Challenge, it demonstrates effectiveness, robustness, and scalability through ablation studies.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We introduce NeuCo-Bench, a novel benchmark framework for evaluating (lossy) neural compression and representation learning in the context of Earth Observation (EO). Our approach builds on fixed-size embeddings that act as compact, task-agnostic representations applicable to a broad range of downstream tasks. NeuCo-Bench comprises three core components: (i) an evaluation pipeline built around reusable embeddings, (ii) a new challenge mode with a hidden-task leaderboard designed to mitigate pretraining bias, and (iii) a scoring system that balances accuracy and stability. To support reproducibility, we release SSL4EO-S12-downstream, a curated multispectral, multitemporal EO dataset. We present initial results from a public challenge at the 2025 CVPR EARTHVISION workshop and conduct ablations with state-of-the-art foundation models. NeuCo-Bench provides a first step towards community-driven, standardized evaluation of neural embeddings for EO and beyond.

Problem

• Evaluating neural compression for Earth Observation data
• Creating task-agnostic embeddings for downstream applications
• Establishing standardized benchmark to mitigate pretraining bias

Innovation

• Fixed-size embeddings for task-agnostic representations
• Hidden-task leaderboard to mitigate pretraining bias
• Scoring system balancing accuracy and stability