Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature

2025-11-05

AI Summary

This study systematically evaluates performance disparities among diverse large language models (LLMs) in retrieval-augmented generation (RAG) question answering over computer science literature. We benchmark both open-source (Mistral-7B-Instruct, Orca-mini-v3-7B, LLaMA2-7B-Chat) and closed-source (GPT-3.5, Gemini) LLMs within a unified RAG framework, measuring accuracy, human evaluation scores, cosine similarity to reference answers, and response latency. Results show that Mistral-7B-Instruct and GPT-3.5 achieve top-tier answer accuracy post-RAG enhancement; Orca-mini-v3-7B exhibits the lowest latency, whereas LLaMA2-7B-Chat incurs the highest; and open-source models collectively attain performance comparable to commercial closed-source counterparts. To our knowledge, this is the first fine-grained empirical analysis of RAG–LLM co-performance specifically on CS literature, revealing fundamental accuracy–latency trade-offs. The study establishes a reproducible, domain-specific benchmark for informed model selection in CS-oriented RAG system design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Retrieval Augmented Generation (RAG) is emerging as a powerful technique to enhance the capabilities of Generative AI models by reducing hallucination. Thus, the increasing prominence of RAG alongside Large Language Models (LLMs) has sparked interest in comparing the performance of different LLMs in question-answering (QA) in diverse domains. This study compares the performance of four open-source LLMs, Mistral-7b-instruct, LLaMa2-7b-chat, Falcon-7b-instruct and Orca-mini-v3-7b, and OpenAI's trending GPT-3.5 over QA tasks within the computer science literature leveraging RAG support. Evaluation metrics employed in the study include accuracy and precision for binary questions and ranking by a human expert, ranking by Google's AI model Gemini, alongside cosine similarity for long-answer questions. GPT-3.5, when paired with RAG, effectively answers binary and long-answer questions, reaffirming its status as an advanced LLM. Regarding open-source LLMs, Mistral AI's Mistral-7b-instruct paired with RAG surpasses the rest in answering both binary and long-answer questions. However, among the open-source LLMs, Orca-mini-v3-7b reports the shortest average latency in generating responses, whereas LLaMa2-7b-chat by Meta reports the highest average latency. This research underscores the fact that open-source LLMs, too, can go hand in hand with proprietary models like GPT-3.5 with better infrastructure.

Problem

• Evaluating LLM performance in RAG-based question-answering systems
• Comparing proprietary and open-source models on computer science literature
• Assessing accuracy, latency, and effectiveness with RAG integration

Innovation

• Evaluates LLMs using Retrieval Augmented Generation technique
• Compares performance across multiple open-source and proprietary models
• Assesses models on accuracy, precision, and response latency