Learning More Effective Representations for Dense Retrieval through Deliberate Thinking Before Search

2025-02-18

AI Summary

To address the limitation of single-embedding document representations in dense retrieval—namely, their inability to capture multi-faceted semantic aspects—this paper proposes DEBATER, a novel framework featuring Chain-of-Deliberation: a first-of-its-kind mechanism that iteratively refines document representations through multi-step reasoning. DEBATER further incorporates Self-Distillation to consolidate critical reasoning steps into a unified, robust, and information-rich strong embedding. By deeply integrating large language model capabilities, chain-of-thought reasoning, and self-distillation, DEBATER achieves significant improvements over state-of-the-art dense retrievers (e.g., ColBERT, ANCE) on major benchmarks including MSMARCO and BEIR, notably enhancing Recall@10 and MRR. Moreover, it demonstrates superior generalization and robustness across diverse domains and query distributions. The implementation is publicly available.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Distributed Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent dense retrievers usually thrive on the emergency capabilities of Large Language Models (LLMs), using them to encode queries and documents into an embedding space for retrieval. These LLM-based dense retrievers have shown promising performance across various retrieval scenarios. However, relying on a single embedding to represent documents proves less effective in capturing different perspectives of documents for matching. In this paper, we propose Deliberate Thinking based Dense Retriever (DEBATER), which enhances these LLM-based retrievers by enabling them to learn more effective document representations through a step-by-step thinking process. DEBATER introduces the Chain-of-Deliberation mechanism to iteratively optimize document representations using a continuous chain of thought. To consolidate information from various thinking steps, DEBATER also incorporates the Self Distillation mechanism, which identifies the most informative thinking steps and integrates them into a unified text embedding. Experimental results show that DEBATER significantly outperforms existing methods across several retrieval benchmarks, demonstrating superior accuracy and robustness. All codes are available at https://github.com/OpenBMB/DEBATER.

Problem

• Enhancing dense retrieval effectiveness
• Optimizing document representations iteratively
• Integrating informative thinking steps

Innovation

• Chain-of-Deliberation mechanism optimizes representations
• Self Distillation integrates informative thinking steps
• DEBATER enhances LLM-based retrievers effectively