When should I search more: Adaptive Complex Query Optimization with Reinforcement Learning

2026-01-29

AI Summary

This work addresses the challenge of effectively handling complex queries in retrieval-augmented generation (RAG) systems, where single-step retrieval often proves insufficient and existing reinforcement learning approaches suffer from unstable training due to combinatorial search spaces and difficulty in reward design. To overcome these limitations, the authors propose the Adaptive Complex Query Optimization (ACQO) framework, which integrates adaptive query decomposition, a ranking-scoring fusion module, and curriculum-based reinforcement learning. ACQO dynamically determines when to decompose queries and how deeply to perform multi-path retrieval, enabling stable and efficient optimization. The framework is compatible with diverse retrieval architectures and achieves state-of-the-art performance across three complex query benchmarks, significantly outperforming existing methods while simultaneously improving computational efficiency and generalization capability.

Technology Category

• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Query optimization is a crucial component for the efficacy of Retrieval-Augmented Generation (RAG) systems. While reinforcement learning (RL)-based agentic and reasoning methods have recently emerged as a promising direction on query optimization, most existing approaches focus on the expansion and abstraction of a single query. However, complex user queries are prevalent in real-world scenarios, often requiring multiple parallel and sequential search strategies to handle disambiguation and decomposition. Directly applying RL to these complex cases introduces significant hurdles. Determining the optimal number of sub-queries and effectively re-ranking and merging retrieved documents vastly expands the search space and complicates reward design, frequently leading to training instability. To address these challenges, we propose a novel RL framework called Adaptive Complex Query Optimization (ACQO). Our framework is designed to adaptively determine when and how to expand the search process. It features two core components: an Adaptive Query Reformulation (AQR) module that dynamically decides when to decompose a query into multiple sub-queries, and a Rank-Score Fusion (RSF) module that ensures robust result aggregation and provides stable reward signals for the learning agent. To mitigate training instabilities, we adopt a Curriculum Reinforcement Learning (CRL) approach, which stabilizes the training process by progressively introducing more challenging queries through a two-stage strategy. Our comprehensive experiments demonstrate that ACQO achieves state-of-the-art performance on three complex query benchmarks, significantly outperforming established baselines. The framework also showcases improved computational efficiency and broad compatibility with different retrieval architectures, establishing it as a powerful and generalizable solution for next-generation RAG systems.

Problem

• Query Optimization
• Retrieval-Augmented Generation
• Complex Queries
• Reinforcement Learning
• Search Strategy

Innovation

• Adaptive Query Reformulation
• Rank-Score Fusion
• Curriculum Reinforcement Learning
• Complex Query Optimization
• Retrieval-Augmented Generation