O1 Embedder: Let Retrievers Think Before Action

2025-02-11

AI Summary

Existing dense retrieval models lack interpretable, step-by-step reasoning—especially in multi-task, zero-shot, and relation-intensive retrieval scenarios. Method: We propose a “reasoning-first” dense retrieval paradigm: before generating document embeddings, the retriever first produces structured intermediate reasoning traces—termed “retrieval thoughts.” We integrate large language models’ (LLMs) progressive reasoning capability into embedding learning via a joint optimization framework unifying retrieval-thought generation and dense retrieval. To train this framework, we introduce a data synthesis strategy combining LLM-based expert initialization with refinement by a retrieval committee, followed by joint fine-tuning via behavior cloning and contrastive learning. Contribution/Results: Our approach achieves significant accuracy improvements across 12 cross-domain benchmark retrieval datasets, demonstrating strong generalization and task adaptability without task-specific architecture modifications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The growing power of large language models (LLMs) has revolutionized how people access and utilize information. Notably, the LLMs excel at performing fine-grained data representation, which facilitates precise retrieval of information. They also generate high-quality answers based on external references, enabling the production of useful knowledge. The recent introduction of reasoning models, like OpenAI O1 and DeepSeek R1, marks another leap forward, highlighting LLMs' ability to think progressively before delivering final answers. This breakthrough significantly improves the ability to address complex tasks, e.g., coding and math proofs. Inspired by this progress, we aim to develop similar capabilities for retrieval models, which hold great promise for tackling critical challenges in the field, including multi-task retrieval, zero-shot retrieval, and tasks requiring intensive reasoning of complex relationships. With this motivation, we propose a novel approach called O1 Embedder, which generates useful thoughts for the input query before making retrieval for the target documents. To realize this objective, we conquer two technical difficulties. First, we design a data synthesis workflow, creating training signals for O1 Embedder by generating initial thoughts from an LLM-expert and subsequently refining them using a retrieval committee. Second, we optimize the training process, enabling a pre-trained model to be jointly fine-tuned to generate retrieval thoughts via behavior cloning and perform dense retrieval through contrastive learning. Our approach is evaluated by comprehensive experiments, where substantial improvements are achieved across 12 popular datasets, spanning both in-domain and out-of-domain scenarios. These results highlight O1 Embedder's remarkable accuracy and generalizability, paving the way for the development of next-generation IR foundation models.

Problem

• Enhance retrieval models' reasoning capabilities
• Address multi-task and zero-shot retrieval challenges
• Improve accuracy and generalizability in information retrieval

Innovation

• LLM-based retrieval enhancement
• multi-task reasoning optimization
• contrastive learning fine-tuning