Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval

2024-12-21

arXiv.org

AI Summary

This work addresses implicit-reasoning retrieval—a challenging task where queries and relevant documents lack direct semantic similarity and instead require inference via logical chains, causal relationships, or experiential knowledge. We propose the first framework leveraging large language models (LLMs) as implicit-reasoning retrievers. Methodologically, we design a generative binary-classification instruction template and integrate it with a cross-encoder architecture, fine-tuned efficiently via Direct Preference Optimization (DPO). The approach achieves strong zero-shot generalization and supervised fine-tuning performance. Evaluated on the Emotional Support Conversation (ESC) benchmark, our model significantly outperforms existing retrieval methods, demonstrating both reasoning effectiveness and robustness. To foster reproducibility and community advancement, we publicly release all code, model weights, and annotated datasets—establishing a foundational resource for implicit-reasoning retrieval research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Despite the recent advancement in Retrieval-Augmented Generation (RAG) systems, most retrieval methodologies are often developed for factual retrieval, which assumes query and positive documents are semantically similar. In this paper, we instead propose and study a more challenging type of retrieval task, called hidden rationale retrieval, in which query and document are not similar but can be inferred by reasoning chains, logic relationships, or empirical experiences. To address such problems, an instruction-tuned Large language model (LLM) with a cross-encoder architecture could be a reasonable choice. To further strengthen pioneering LLM-based retrievers, we design a special instruction that transforms the retrieval task into a generative task by prompting LLM to answer a binary-choice question. The model can be fine-tuned with direct preference optimization (DPO). The framework is also optimized for computational efficiency with no performance degradation. We name this retrieval framework by RaHoRe and verify its zero-shot and fine-tuned performance superiority on Emotional Support Conversation (ESC), compared with previous retrieval works. Our study suggests the potential to employ LLM as a foundation for a wider scope of retrieval tasks. Our codes, models, and datasets are available on https://github.com/flyfree5/LaHoRe.

Problem

• Retrieving documents with hidden rationales, not semantic similarity
• Using LLMs for reasoning-based retrieval tasks
• Enhancing retrieval efficiency without performance loss

Innovation

• Instruction-tuned LLM with cross-encoder architecture
• Transforms retrieval into generative binary-choice task
• Optimized with DPO for computational efficiency