🤖 AI Summary
This study systematically evaluates the effectiveness and efficiency of hybrid retrieval and re-ranking strategies in Retrieval-Augmented Generation (RAG) across varying scales of scientific corpora—specifically 1K, 5K, and 15K CORD-19 papers. The approach integrates sentence-window chunking, BM25 and BGE-M3 dense retrieval, reciprocal rank fusion, optional cross-encoder re-ranking, and evidence-based answer generation. Results demonstrate that hybrid retrieval achieves perfect Recall@10 (1.000) at both the 1K and 15K corpus scales, while RAGAS faithfulness improves with larger corpus sizes. Notably, cross-encoders trained on general-domain data degrade accuracy due to domain mismatch. The work introduces a controllable evaluation paradigm based on pseudo relevance labels, revealing the superior performance of hybrid retrieval for scientific question answering.
📝 Abstract
We introduce SciRet, a compute-aware empirical study of retrieval-augmented generation for scientific question answering over CORD-19. Rather than proposing a new model, we evaluate a fixed scientific RAG pipeline across three corpus scales: 1,034 chunks (1K papers), 5,160 chunks (5K papers), and 15,480 chunks (15K papers). The pipeline combines sentence-window chunking, BM25, BGE-M3 dense retrieval, reciprocal rank fusion, optional cross-encoder reranking, and grounded answer generation. Across these settings, hybrid retrieval is more robust than either sparse-only or dense-only retrieval in our setting, reaching Recall@10 of 1.000 at 1K and 15K. In contrast, an MS MARCO-trained cross-encoder reranker reduces precision on the scientific corpus, suggesting that domain mismatch can outweigh the benefits of stronger query-passage interaction. Generation faithfulness measured with RAGAS increases with corpus scale in our setup. Retrieval evaluation uses pseudo-relevance labels derived from the hybrid system, so we treat the results as controlled comparative evidence rather than a benchmark claim. We release code, indexes, and evaluation outputs to support replication and follow-up studies.