Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach

2024-07-18

arXiv.org

AI Summary

Multi-hop question answering faces challenges including difficulty in acquiring all necessary evidence in a single retrieval step, information overload from iterative retrieval, and lack of explicit process tracking. This paper proposes ReSP, an iterative retrieval-augmented generation framework. ReSP introduces a novel dual-objective summarizer that concurrently compresses information relevant to both the global question and dynamically generated sub-questions. It incorporates a retrieval trajectory memory mechanism to explicitly model and record retrieval paths, thereby suppressing redundant planning. Additionally, it leverages LLM-driven sub-question decomposition and verification for lightweight, efficient iterative information integration. Evaluated on HotpotQA and 2WikiMultihopQA, ReSP achieves significant improvements over state-of-the-art methods, demonstrates strong robustness to long contexts, improves inference efficiency by 23%, and attains a summary compression rate of 68%.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Question Answering
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multi-hop question answering is a challenging task with distinct industrial relevance, and Retrieval-Augmented Generation (RAG) methods based on large language models (LLMs) have become a popular approach to tackle this task. Owing to the potential inability to retrieve all necessary information in a single iteration, a series of iterative RAG methods has been recently developed, showing significant performance improvements. However, existing methods still face two critical challenges: context overload resulting from multiple rounds of retrieval, and over-planning and repetitive planning due to the lack of a recorded retrieval trajectory. In this paper, we propose a novel iterative RAG method called ReSP, equipped with a dual-function summarizer. This summarizer compresses information from retrieved documents, targeting both the overarching question and the current sub-question concurrently. Experimental results on the multi-hop question-answering datasets HotpotQA and 2WikiMultihopQA demonstrate that our method significantly outperforms the state-of-the-art, and exhibits excellent robustness concerning context length.

Problem

• Multi-hop Question Answering
• Information Overload
• Clue Tracking

Innovation

• ReSP Method
• Information Extraction
• Multi-hop Question Answering