🤖 AI Summary
This study investigates the trade-offs among generation quality, inference latency, and computational resources in retrieval-augmented generation (RAG) systems. Leveraging a human-curated benchmark of 5,144 Kubernetes question-answer pairs and a hybrid retrieval pipeline—combining BGE-M3 dense and sparse vectors, reciprocal rank fusion (RRF), and cross-encoder reranking—the authors systematically evaluate 20 LoRA configurations on Llama-3.1-8B and Llama-3.2-3B. For the first time, they quantitatively characterize LoRA’s multi-dimensional trade-offs in RAG settings, demonstrating that LoRA variants adapting only the query and value projections consistently dominate the Pareto frontier, with this advantage stemming from architectural design rather than parameter count. Evaluation metrics include token-level F1, LLM-judged groundedness and correctness (pass@4), latency, memory usage, and training cost. The dataset, code, and optimized adapters are publicly released.
📝 Abstract
We study quality-latency-resource trade-offs in a documentation-grounded retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA) of the generator. We build a manually verified benchmark of 5,144 question-answer pairs over the official Kubernetes documentation and combine it with a fixed hybrid-retrieval pipeline (BGE-M3 dense, BGE-M3 native sparse, Reciprocal Rank Fusion, cross-encoder reranking). Over this benchmark we ablate 20 LoRA configurations on Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct across rank and target-module choices, and evaluate each on token-level F1, LLM-judged groundedness and correctness (pass@4), inference latency, inference memory, and training cost, all reported with bootstrap 95% confidence intervals. Pareto analysis shows that LoRA adapters acting only on the q and v attention projections consistently dominate the front, while the 3B/8B choice mainly defines operating regime. A param-matched control comparison further indicates that the q/v advantage is structural rather than purely parametric. The benchmark, selected adapters, and code are available at https://github.com/EugPal/rag-lora-tradeoffs.