Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning

2026-01-13

AI Summary

This work addresses the challenge of hallucination in tool-augmented retrieval-augmented generation (agentic RAG) systems, where user queries often fall outside the system’s knowledge scope. While existing approaches merely filter out invalid questions, they lack mechanisms to guide users toward answerable formulations. To bridge this gap, this study introduces query suggestion into the agentic RAG paradigm for the first time, proposing an answerability-aware dynamic in-context learning method. By retrieving relevant workflow examples from historical interactions, the approach constructs few-shot prompts that enable a large language model to generate semantically similar yet answerable reformulations of the original query. The method supports multi-step workflows and incorporates self-learning capabilities. Evaluated on three real-world user query benchmarks, it significantly outperforms conventional few-shot and pure retrieval baselines, yielding suggestions that exhibit both high relevance and strong answerability.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Retrieval-augmented generation with tool-calling agents (agentic RAG) has become increasingly powerful in understanding, processing, and responding to user queries. However, the scope of the grounding knowledge is limited and asking questions that exceed this scope may lead to issues like hallucination. While guardrail frameworks aim to block out-of-scope questions (Rodriguez et al., 2024), no research has investigated the question of suggesting answerable queries in order to complete the user interaction. In this paper, we initiate the study of query suggestion for agentic RAG. We consider the setting where user questions are not answerable, and the suggested queries should be similar to aid the user interaction. Such scenarios are frequent for tool-calling LLMs as communicating the restrictions of the tools or the underlying datasets to the user is difficult, and adding query suggestions enhances the interaction with the RAG agent. As opposed to traditional settings for query recommendations such as in search engines, ensuring that the suggested queries are answerable is a major challenge due to the RAG's multi-step workflow that demands a nuanced understanding of the RAG as a whole, which the executing LLM lacks. As such, we introduce robust dynamic few-shot learning which retrieves examples from relevant workflows. We show that our system can be self-learned, for instance on prior user queries, and is therefore easily applicable in practice. We evaluate our approach on three benchmark datasets based on two unlabeled question datasets collected from real-world user queries. Experiments on real-world datasets confirm that our method produces more relevant and answerable suggestions, outperforming few-shot and retrieval-only baselines, and thus enable safer, more effective user interaction with agentic RAG.

Problem

• query suggestion
• retrieval-augmented generation
• agentic RAG
• answerable queries
• user interaction

Innovation

• query suggestion
• retrieval-augmented generation
• dynamic in-context learning
• tool-calling agents
• answerable queries