REDSI: Addressing the Reproducibility and Evaluation Consistency of Differentiable Search Indexing for Document Retrieval

2026-09-08

AI Summary

本文通过引入ReDSI，一个支持所有三种文档标识符类型的开源实现及规范化的NQ320K构建流程，解决了不同可微搜索索引框架的再现性和评估一致性问题。

Abstract

The differentiable search index (DSI) framework (Tay et al., 2022) has become the de facto baseline for generative retrieval. However, DSI is hard to reproduce: no public implementation covers all three original document identifier types (atomic, naive, semantic), reported results vary widely, and the ubiquitous NQ320K dataset is built from Natural Questions through diverse and underspecified preprocessing. We introduce ReDSI, the first open-source DSI implementation supporting all three identifier types, together with a parameterizable and well-documented NQ320K construction pipeline. Experimentally, we achieve results that are competitive with or stronger than previous DSI baselines. Moreover, we conduct extensive experiments under model downscaling, covering retrieval effectiveness, parameter efficiency, training methods and decoding strategies, opening novel directions for future research.

Problem

• Differentiable Search Index
• Reproducibility
• Evaluation Consistency
• Document Retrieval
• NQ320K

Innovation

• ReDSI
• differentiable search index
• document retrieval
• NQ320K dataset
• reproducibility