Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation

2024-07-01

arXiv.org

AI Summary

To address redundancy, intent deviation, and incomplete condition coverage in Coverage-Conditioned (C²) queries, this paper proposes QPlanner—a novel framework that first formalizes structured subquery outline generation as an evaluable and optimizable coverage control task. Methodologically, we introduce QTree, the first hierarchical benchmark comprising 10K subqueries; design a lightweight QPlanner model (7B); and integrate supervised fine-tuning on QTree with Direct Preference Optimization (DPO)-based alignment training, embedded within a RAG pipeline. Key contributions include: (1) a coverage-aware outline generation paradigm, and (2) a multidimensional evaluation framework combining automated metrics and human judgment. Experiments demonstrate that QPlanner significantly improves RAG response relevance, coverage completeness, and user satisfaction—achieving +18.7% gain over baselines on automated C² metrics and +23% on human evaluations.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Web Mining and Content Analysis: Community question answering

Abstract

Interactions with large language models (LLMs) often yield long and detailed responses, leveraging both parametric knowledge and retrieval-augmented generation (RAG). While these responses can provide rich insights, they often include redundant or less engaging content not aligned with user interests. This issue becomes apparent when users specify particular subtopics to include or exclude -- termed coverage-conditioned ($C^2$) queries -- as LLMs often struggle to provide tailored responses. To address this challenge, we investigate the role of query outlines, sequences of subqueries designed to guide LLMs in generating responses that meet specific user requirements. To systematically create and evaluate these outlines, we introduce QTree, a dataset of 10K hierarchical sets of information-seeking subqueries that define structured boundaries for outline creation and evaluation in $C^2$ scenarios. Additionally, we develop QPlanner, a 7B language model trained to generate customized outlines within boundaries of QTree. We evaluate the effectiveness of the generated outlines through automatic and human judgements, focusing on their impact within retrieval-augmented generation (RAG) systems. Experimental results demonstrate that QPlanner, especially when trained with alignment techniques like DPO, generates higher-quality outlines that better fulfill diverse user needs.

Problem

• Information Redundancy
• Irrelevant Content
• Conditional Response Limitations

Innovation

• QTree
• QPlanner
• Structured Query Framework