DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers

2025-05-28

AI Summary

In multimodal RAG, reranker training is hindered by low-quality hard negatives—exhibiting insufficient diversity, inadequate difficulty, frequent false negatives, and poor controllability. To address this, we propose a “page-centric” paradigm for generating single-page hard negative queries: given a document page and its positive queries, we employ a collaborative LLM–VLM pipeline to synthesize semantically similar yet unanswerable queries—i.e., queries the page cannot answer. Our approach integrates query rephrasing modeling, multimodal semantic consistency constraints, and automated false-negative verification. This framework significantly enhances the hardness, diversity, and authenticity of hard negatives while enabling fine-grained, controllable generation. Evaluated on established multimodal RAG benchmarks, the reranker trained with our negatives achieves substantial improvements in retrieval accuracy and robustness over state-of-the-art methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Rerankers play a critical role in multimodal Retrieval-Augmented Generation (RAG) by refining ranking of an initial set of retrieved documents. Rerankers are typically trained using hard negative mining, whose goal is to select pages for each query which rank high, but are actually irrelevant. However, this selection process is typically passive and restricted to what the retriever can find in the available corpus, leading to several inherent limitations. These include: limited diversity, negative examples which are often not hard enough, low controllability, and frequent false negatives which harm training. Our paper proposes an alternative approach: Single-Page Hard Negative Query Generation, which goes the other way around. Instead of retrieving negative pages per query, we generate hard negative queries per page. Using an automated LLM-VLM pipeline, and given a page and its positive query, we create hard negatives by rephrasing the query to be as similar as possible in form and context, yet not answerable from the page. This paradigm enables fine-grained control over the generated queries, resulting in diverse, hard, and targeted negatives. It also supports efficient false negative verification. Our experiments show that rerankers trained with data generated using our approach outperform existing models and significantly improve retrieval performance.

Problem

• Generates hard negative queries per page for RAG rerankers
• Improves diversity and hardness of negative training examples
• Enables fine-grained control and reduces false negatives

Innovation

• Generates hard negative queries per page
• Uses LLM-VLM pipeline for query rephrasing
• Ensures diverse and controllable negative examples