A Reranker for Orchestrating Heterogeneous Speech and Text Retrievers

2026-08-24

AI Summary

本文提出STeReO，一种基于语音和文本检索器的重排序器，通过聚合不同模态数据库解决多模态场景下证据选择问题，提高问答性能。

Abstract

Retrieval-Augmented Generation (RAG) systems have attracted significant interest for their ability to mitigate hallucinations in Large Language Models (LLMs). Although knowledge databases for RAG are increasingly diversifying to include various modalities such as speech and text, research on handling such multi-modal database scenarios remains limited. In this paper, we propose STeReO (Speech and Text Reranking Orchestrator), a reranker based on speech and text retrievers that aggregates disparate modality databases. To address the lack of specialized training data, we first curate a dataset comprising queries, mixed-modality evidence, and their corresponding relevance ranks. We then train the reranker and evaluate its effectiveness in both single-modality and mixed-modality scenarios. Our results demonstrate that the proposed algorithm excels at selecting the most relevant evidence, thereby significantly improving downstream question-answering performance.

Problem

• Retrieval-Augmented Generation
• multi-modal database
• relevance ranks
• question-answering performance

Innovation

• Reranker
• Multi-modal Database
• Retrieval-Augmented Generation
• Mixed-modality Evidence
• Question-Answering Performance