Understanding and Debugging Failures in N-Gram-Based Generative Retrieval

2026-06-16

AI Summary

This study systematically investigates critical failure mechanisms in N-gram-based generative retrieval approaches, such as SEAL and MINDER, including document identifier ambiguity, insufficient diversity, and the disproportionate influence of individual identifiers on ranking outcomes. To address these issues, we introduce the first comprehensive taxonomy of failure modes specific to generative retrieval and employ empirical analysis combined with error attribution techniques to dissect flaws in the N-gram generation process. Furthermore, we develop an open-source, interactive visualization tool that enables researchers to intuitively diagnose how identifier generation impacts retrieval performance. Our work substantially enhances the interpretability and controllability of N-gram-based generative retrieval systems, offering both a theoretical foundation and practical resources to advance this emerging paradigm.

Abstract

Generative Retrieval (GR) is an emerging Information Retrieval (IR) paradigm that is motivated by increasingly capable language models. In GR, a model directly generates identifiers for relevant documents. While these systems offer unique advantages, they also introduce distinct failure mechanisms. We explore these failure modes in three contributions: (1) We present a taxonomy of GR failure modes based on GR literature. (2) We empirically investigate failure in a subset of GR: ngram-based methods, more specifically, SEAL and MINDER. Our analysis reveals common issues, such as ambiguous docids, low identifier diversity, and the disproportionate impact of specific identifiers. (3) We introduce a new web-based tool that helps the IR community analyze generated ngrams and their respective contribution to the final ranking, providing an intuitive interface to identify where such GR methods go wrong.

Problem

• Generative Retrieval
• failure modes
• n-gram
• document identifiers
• information retrieval

Innovation

• Generative Retrieval
• Failure Analysis
• n-gram-based Methods
• Identifier Ambiguity
• Debugging Tool