Bayesian Active Learning with Gaussian Processes Guided by LLM Relevance Scoring for Dense Passage Retrieval

2026-04-20

AI Summary

This work addresses the challenge that existing dense retrieval methods struggle to effectively recall semantically diverse and relevant passages under a limited LLM query budget and lack global modeling of relevance signals. The authors formalize passage retrieval as a global optimization problem under a budget constraint and propose the first approach integrating Gaussian processes with LLM-based relevance scoring. Specifically, they employ a Gaussian process to model the multimodal relevance distribution in the embedding space, enabling effective propagation of sparse relevance signals, and iteratively select the most informative passages for LLM evaluation via a Bayesian active learning strategy that dynamically balances exploration and exploitation. Experiments demonstrate that the proposed method significantly outperforms LLM-based reranking baselines under equivalent query budgets across four benchmark datasets and two LLM backbones, achieving superior retrieval performance in scenarios with complex relevance structures.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While Large Language Models (LLMs) exhibit exceptional zero-shot relevance modeling, their high computational cost necessitates framing passage retrieval as a budget-constrained global optimization problem. Existing approaches passively rely on first-stage dense retrievers, which leads to two limitations: (1) failing to retrieve relevant passages in semantically distinct clusters, and (2) failing to propagate relevance signals to the broader corpus. To address these limitations, we propose Bayesian Active Learning with Gaussian Processes guided by LLM relevance scoring (BAGEL), a novel framework that propagates sparse LLM relevance signals across the embedding space to guide global exploration. BAGEL models the multimodal relevance distribution across the entire embedding space with a query-specific Gaussian Process (GP) based on LLM relevance scores. Subsequently, it iteratively selects passages for scoring by strategically balancing the exploitation of high-confidence regions with the exploration of uncertain areas. Extensive experiments across four benchmark datasets and two LLM backbones demonstrate that BAGEL effectively explores and captures complex relevance distributions and outperforms LLM reranking methods under the same LLM budget on all four datasets.

Problem

• dense passage retrieval
• relevance propagation
• budget-constrained optimization
• semantic clustering
• LLM relevance scoring

Innovation

• Bayesian Active Learning
• Gaussian Processes
• LLM Relevance Scoring
• Dense Passage Retrieval
• Budget-Constrained Optimization