MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

2026-06-23

AI Summary

Existing benchmarks inadequately evaluate the synergistic capabilities of cross-lingual alignment, conceptual disambiguation, and evidence retrieval in multilingual medical information retrieval, exhibiting significant performance degradation in non-English settings. This work introduces the first multilingual medical retrieval benchmark spanning six languages, featuring three decoupled tasks—cross-lingual question answering retrieval, multi-level concept discrimination, and RAG-oriented evidence retrieval—with no overlap in queries or concepts across tasks. Leveraging a UMLS-based cross-lingual query set, a three-tier confusable concept set, and a controlled RAG task design, the study systematically evaluates ten mainstream models. Results reveal a drastic drop in nDCG@10 from 0.818 to 0.056 for languages such as Japanese, underscoring a critical deficiency in current methods’ cross-lingual generalization capacity.

Abstract

Retrieval-augmented generation (RAG) in clinical settings increasingly requires multilingual retrieval against predominantly English evidence corpora. Multilingual medical retrieval demands three capabilities: cross-lingual alignment, concept discrimination, and evidence retrieval. However, existing benchmarks evaluate these only in isolation, leaving the interaction between biomedical expertise and multilingual coverage unmeasured. We introduce MMed-Bench-IR, a benchmark designed to disentangle these axes across 6 languages and three structurally heterogeneous tasks: (1) cross-lingual medical QA retrieval with 6,127 queries grounded in the Unified Medical Language System (UMLS), (2) concept discrimination over 4,975 confusion sets at three difficulty tiers, and (3) multilingual evidence retrieval for RAG with 2,040 quality-assured queries. The three tasks share zero concept and query overlap by design, ensuring that aggregate scores reflect genuine capability breadth. Evaluation of ten systems across six paradigm families reveals severe cross-lingual failure: biomedical encoders that score 0.818 nDCG@10 in English drop to 0.056 in Japanese, a gap that English-only benchmarks cannot detect.

Problem

• multilingual medical information retrieval
• cross-lingual alignment
• concept discrimination
• evidence retrieval
• benchmark evaluation

Innovation

• multilingual medical retrieval
• heterogeneous benchmark
• cross-lingual alignment
• concept discrimination
• retrieval-augmented generation