Rethinking Agentic RAG: Toward LLM-Driven Logical Retrieval Beyond Embeddings

2026-05-26

AI Summary

This work addresses the overreliance of existing agent-based RAG systems on complex retrieval backends, which constrains the ability of large language models (LLMs) to precisely articulate information needs. The authors propose a novel agent RAG framework that restores retrieval control to the LLM by enabling it to explicitly formulate query intent through generated logical expressions, coupled with a lightweight inverted index for structured retrieval. By eschewing sophisticated embedding mechanisms, the approach substantially simplifies system architecture while maintaining performance comparable to strong baselines. This design significantly reduces both construction and serving costs and effectively mitigates hallucination in generated outputs.

Abstract

Recent advances in RAG have shifted toward an agentic paradigm, where LLMs interact with retrieval systems over multiple turns and iteratively refine queries based on intermediate results. At the same time, LLMs have demonstrated a strong ability to construct structured queries that precisely express their information needs. However, contemporary RAG systems remain heavily focused on engineering complex retrieval backends, including dense, hybrid, and graph-based retrieval architectures. In this study, we argue that agentic RAG should delegate greater control to the LLM to steer the retrieval process, while relying on a lightweight retrieval interface that provides fine-grained control and faithfully executes the LLM's structured intent. Guided by this principle, we propose an agentic RAG framework that enables LLMs to formulate retrieval intents using logical expressions while simplifying the retrieval backend to an inverted-index-based system. Extensive experiments show that our framework matches a strong agentic hybrid baseline, while substantially reducing construction and serving cost. Moreover, we show that anchoring the retrieval process in logical queries substantially reduces hallucinations in generated responses.

Problem

• Agentic RAG
• LLM-driven retrieval
• logical queries
• retrieval backend complexity
• hallucination reduction

Innovation

• Agentic RAG
• Logical Retrieval
• LLM-driven Querying
• Inverted Index
• Hallucination Reduction