Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach

2026-02-14

AI Summary

This study addresses the limited reasoning capabilities of small language models (SLMs) in multi-hop question answering by systematically evaluating 24 prompt templates on the HotpotQA dataset. The evaluation encompasses standard RAG prompts, nine existing prompting strategies, and fourteen newly designed hybrid prompts, with experiments conducted on Qwen2.5-3B and Gemma3-4B-It. The work introduces the first efficient hybrid prompting template tailored for SLMs, significantly enhancing multi-hop reasoning performance under resource-constrained conditions. On a test set of 18,720 samples, the proposed approach achieves up to 83% and 84.5% relative accuracy improvements over standard RAG prompts for the two models, respectively, corresponding to an absolute accuracy gain of up to 6%. The paper also provides a reproducible guideline for effective prompt design in SLM-based multi-hop QA systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Retrieval Augmented Generation (RAG) is a powerful approach for enhancing the factual grounding of language models by integrating external knowledge. While widely studied for large language models, the optimization of RAG for Small Language Models (SLMs) remains a critical research gap, particularly in complex, multi-hop question-answering tasks that require sophisticated reasoning. In these systems, prompt template design is a crucial yet under-explored factor influencing performance. This paper presents a large-scale empirical study to investigate this factor, evaluating 24 different prompt templates on the HotpotQA dataset. The set includes a standard RAG prompt, nine well-formed techniques from the literature, and 14 novel hybrid variants, all tested on two prominent SLMs: Qwen2.5-3B Instruct and Gemma3-4B-It. Our findings, based on a test set of 18720 instances, reveal significant performance gains of up to 83% on Qwen2.5 and 84.5% on Gemma3-4B-It, yielding an improvement of up to 6% for both models compared to the Standard RAG prompt. This research also offers concrete analysis and actionable recommendations for designing effective and efficient prompts for SLM-based RAG systems, practically for deployment in resource-constrained environments.

Problem

• Retrieval Augmented Generation
• Small Language Models
• Prompt Engineering
• Multi-Hop QA

Innovation

• Prompt Engineering
• Retrieval Augmented Generation (RAG)
• Small Language Models (SLMs)
• Multi-Hop Question Answering
• Hybrid Prompt Templates