NeuCLIRTech: Chinese Monolingual and Cross-Language Information Retrieval Evaluation in a Challenging Domain

2026-02-05

AI Summary

This work addresses the scarcity of high-discriminability monolingual and cross-lingual evaluation benchmarks for technical-domain information retrieval in Chinese. To this end, we introduce NeuCLIRTech, a novel benchmark comprising 110 technical queries with relevance judgments over 35,962 Chinese and English documents, supporting both Chinese monolingual and English-to-Chinese cross-lingual retrieval. NeuCLIRTech provides the first high-quality, bilingual-aligned technical document collection for IR evaluation, integrating TREC NeuCLIR 2023–2024 topics to enhance system discriminability and leveraging machine translation to construct parallel document sets. We propose a neural retrieval fusion baseline that replaces traditional BM25 initial ranking and incorporates multi-system reranking strategies. The dataset and models are publicly released on Hugging Face Datasets, offering a statistically robust and reliable benchmark to advance both cross-lingual and monolingual retrieval research, particularly in reranking methodologies.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Measuring advances in retrieval requires test collections with relevance judgments that can faithfully distinguish systems. This paper presents NeuCLIRTech, an evaluation collection for cross-language retrieval over technical information. The collection consists of technical documents written natively in Chinese and those same documents machine translated into English. It includes 110 queries with relevance judgments. The collection supports two retrieval scenarios: monolingual retrieval in Chinese, and cross-language retrieval with English as the query language. NeuCLIRTech combines the TREC NeuCLIR track topics of 2023 and 2024. The 110 queries with 35,962 document judgments provide strong statistical discriminatory power when trying to distinguish retrieval approaches. A fusion baseline of strong neural retrieval systems is included so that developers of reranking algorithms are not reliant on BM25 as their first stage retriever. The dataset and artifacts are released on Huggingface Datasets

Problem

• cross-language information retrieval
• Chinese monolingual retrieval
• test collection
• relevance judgments
• technical domain

Innovation

• cross-language information retrieval
• neural retrieval
• evaluation collection
• technical domain
• relevance judgments