Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context

2024-10-09

arXiv.org

AI Summary

Large language models (LLMs) exhibit significant sensitivity to context misordering in multi-hop reasoning—specifically, they are highly dependent on both the absolute positions and relative ordering of supporting documents and struggle to filter out irrelevant information. This paper formally defines the “misordered-context” problem for the first time and proposes Context Repetition (CoRe), a parameter-free, fine-tuning-free prompting paradigm that enhances robustness to reasoning-chain order by strategically repeating key contextual fragments. CoRe is seamlessly integrated into prompt design and fully compatible with chain-of-thought and retrieval-augmented inference. Empirical evaluation demonstrates substantial improvements: up to +30 percentage points in F1 on multi-hop question answering benchmarks, +70 percentage points in accuracy on synthetic controllable reasoning tasks, and marked mitigation of the “intermediate fact loss” phenomenon.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multi-hop reasoning, which requires multi-step reasoning based on the supporting documents within a given context, remains challenging for large language models (LLMs). LLMs often struggle to filter out irrelevant documents within the context, and their performance is sensitive to the absolute position of supporting documents within that context. In this paper, we identify an additional challenge: LLMs' performance is also sensitive to the order, relative position, in which the supporting documents are presented. We refer to this as the misordered context problem. To address this issue, based on the theoretical approach, we propose a simple yet effective method called context repetition (CoRe), which involves prompting the model by repeatedly presenting the context. This ensures that certain contiguous reasoning segments within supporting documents are presented in the optimal order, effectively guiding the model's reasoning in the appropriate direction. Applying CoRe, we improve the F1 score by up to 30%p on multi-hop QA tasks and increase accuracy by up to 70%p on a synthetic task. Additionally, CoRe helps mitigate the well-known"lost-in-the-middle"problem in LLMs and can be effectively combined with retrieval-based approaches utilizing Chain-of-Thought (CoT) reasoning.

Problem

• LLMs struggle with multi-hop reasoning due to irrelevant documents.
• LLM performance is sensitive to document order and position.
• Misordered context problem hinders effective multi-step reasoning in LLMs.

Innovation

• Repeats context to optimize document order
• Improves multi-hop reasoning with CoRe method
• Combines with Chain-of-Thought for better accuracy