Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark

2026-07-03

AI Summary

This work addresses the lack of code search benchmarks tailored to the terminology and requirements of scientific computing, which hinders researchers’ ability to efficiently discover relevant tools. We present the first high-quality, expert-annotated open-source code corpus spanning five NASA scientific domains and introduce two novel retrieval benchmarks: an expert-designed repository-level search task and a large-scale multilingual code snippet retrieval task. Leveraging techniques including repository crawling, README cleaning, topic extraction, and cross-lingual code parsing, we release a public dataset comprising 5,264 repositories, 117,950 code snippets, and 119,720 queries on HuggingFace. Evaluation on these benchmarks reveals significant disparities in code discoverability across scientific domains, effectively bridging the gap between general-purpose code search systems and the practical needs of scientific research.

Abstract

Scientists increasingly rely on open-source tools to support their research workflows, yet discovering relevant software among over 600 million GitHub repositories remains challenging. Existing code search benchmarks focus on general software engineering tasks and fail to capture the domain-specific vocabulary and needs of scientific computing. We present a curated corpus of 5,264 high-quality, domain-classified scientific repositories spanning five NASA Science Mission Directorate divisions -- Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological & Physical Sciences -- enriched with cleaned READMEs, extracted topics, and additional context from crawled links. Building on this corpus, we introduce two novel information retrieval benchmarks: (1) a repository search benchmark with 219 expert-curated queries designed by domain scientists, and (2) a large-scale code snippet retrieval benchmark containing 117,950 code snippets and 119,720 queries across seven programming languages. Baseline evaluations on repository search reveal significant performance variation across scientific domains. Code snippet retrieval proves equally challenging, with substantial variation driven by differing documentation practices, coding standards, and programming language conventions across scientific communities. All datasets and benchmarks are publicly released on HuggingFace to support research on scientific tool discovery.

Problem

• scientific code search
• domain-specific software discovery
• code retrieval benchmark
• scientific computing
• open-source repository discovery

Innovation

• scientific code search
• multi-domain dataset
• code retrieval benchmark
• domain-specific software discovery
• open-source scientific repositories