Re-ranking the Context for Multimodal Retrieval Augmented Generation

2025-01-08

AI Summary

To address low context relevance and hallucination risks in the image-text retrieval stage of multimodal RAG, this paper proposes a learnable dynamic relevancy scoring mechanism for re-ranking, replacing fixed top-k truncation with on-demand, adaptive selection of the most contextually relevant multimodal fragments. It introduces, for the first time, a learnable relevance score into multimodal retrieval re-ranking, jointly leveraging CLIP-based cross-modal embeddings and context-aware semantic alignment. Experiments on the COCO dataset demonstrate significant improvements: context relevance increases by 32.7%, generation accuracy rises by 26.4%, and hallucination rate decreases by 41.1%, effectively mitigating cross-modal semantic mismatch.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant entries to user query (e.g., images, documents), and (ii) vision-language models or multi-modal language models like GPT-4o may hallucinate when processing these entries to generate RAG output. In this paper, we aim to address the first challenge, i.e, improving the selection of relevant context from the knowledge-base in retrieval phase of the multi-modal RAG. Specifically, we leverage the relevancy score (RS) measure designed in our previous work for evaluating the RAG performance to select more relevant entries in retrieval process. The retrieval based on embeddings, say CLIP-based embedding, and cosine similarity usually perform poorly particularly for multi-modal data. We show that by using a more advanced relevancy measure, one can enhance the retrieval process by selecting more relevant pieces from the knowledge-base and eliminate the irrelevant pieces from the context by adaptively selecting up-to-$k$ entries instead of fixed number of entries. Our evaluation using COCO dataset demonstrates significant enhancement in selecting relevant context and accuracy of the generated response.

Problem

• Multimodal Information Processing
• Relevance Judgment
• Large Language Models

Innovation

• Adaptive Relevance Scoring
• Multi-modal Information Processing
• Improved RAG Method