From Matching to Generation: A Survey on Generative Information Retrieval

2024-04-23

arXiv.org

AI Summary

Traditional retrieval systems rely on similarity matching, struggling to balance document-level granularity with flexible relevance modeling. To address this, this paper proposes Generative Information Retrieval (GenIR), a novel paradigm leveraging pretrained language models to directly generate document identifiers or user-requested information—shifting from “matching” to “generating.” We present the first systematic survey of GenIR’s two core directions: generative document retrieval and reliable response generation. The survey unifies key research pathways—including parameterized document memory, synergistic internal-external knowledge integration, incremental learning, and trustworthy evaluation. By integrating instruction tuning, retrieval-augmented generation (RAG), knowledge distillation, and multi-granularity evaluation frameworks, we deliver the field’s first comprehensive GenIR survey. We publicly release a curated GitHub repository, explicitly identify critical challenges, and propose an evolution roadmap emphasizing scalability, verifiability, and trustworthiness.

Technology Category

• Natural Language Processing: Generation
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Information Retrieval (IR) systems are crucial tools for users to access information, which have long been dominated by traditional methods relying on similarity matching. With the advancement of pre-trained language models, generative information retrieval (GenIR) emerges as a novel paradigm, attracting increasing attention. Based on the form of information provided to users, current research in GenIR can be categorized into two aspects: extbf{(1) Generative Document Retrieval} (GR) leverages the generative model's parameters for memorizing documents, enabling retrieval by directly generating relevant document identifiers without explicit indexing. extbf{(2) Reliable Response Generation} employs language models to directly generate information users seek, breaking the limitations of traditional IR in terms of document granularity and relevance matching while offering flexibility, efficiency, and creativity to meet practical needs. This paper aims to systematically review the latest research progress in GenIR. We will summarize the advancements in GR regarding model training and structure, document identifier, incremental learning, etc., as well as progress in reliable response generation in aspects of internal knowledge memorization, external knowledge augmentation, etc. We also review the evaluation, challenges and future developments in GenIR systems. This review aims to offer a comprehensive reference for researchers, encouraging further development in the GenIR field. Github Repository: https://github.com/RUC-NLPIR/GenIR-Survey

Problem

• Transition from similarity matching to generative models in IR systems.
• Explore generative document retrieval and reliable response generation techniques.
• Review advancements, challenges, and future directions in generative information retrieval.

Innovation

• Generative models for document retrieval
• Language models for reliable response generation
• Integration of internal and external knowledge