Joint Enhancement of Relational Reasoning for Long-Context LLMs

2025-08-27

AI Summary

Large language models (LLMs) face critical challenges in long-context understanding, including memory constraints, logical inconsistency, frequent hallucinations, and poor interpretability. To address these issues, we propose JERR—a novel graph-structured reasoning framework that integrates summary-driven text chunking, directed acyclic graph (DAG)-based structural modeling, and Monte Carlo tree search (MCTS). JERR explicitly captures cross-paragraph semantic dependencies and reasoning paths, thereby enhancing transparency and controllability. Extensive experiments demonstrate that JERR consistently outperforms state-of-the-art baselines across ROUGE and F1 metrics. Moreover, it achieves the highest score in human evaluation via LLM-Rater, significantly improving accuracy and robustness on long-text question answering, abstractive summarization, and complex multi-step reasoning tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite significant progress, large language models (LLMs) still struggle with long contexts due to memory limitations and their inability to tackle complex and long-context tasks. Additionally, LLMs often suffer from a lack of transparency and are prone to producing hallucinations. To address these challenges, we propose extbf{JERR}, a novel framework designed to enhance long-context comprehension via graph-based reasoning in LLMs. JERR integrates three key components: synopsis extraction, graph construction, and relational reasoning. First, synopsis is extracted by chunking text strategically, allowing the model to summarize and understand information more efficiently. Second, we build a directed acyclic graph (DAG) to resolve redundancy, ensuring logical consistency and clarity. Finally, we incorporate Monte Carlo Tree Search (MCTS) to help the model navigate complex reasoning paths, ensuring more accurate and interpretable outputs. This framework provides a novel solution that enables LLMs to handle extended contexts and complex reasoning tasks with improved reliability and transparency. Experimental results show that JERR consistently outperforms all baselines on the ROUGE and F1 metrics, achieving the highest scores on the LLM-Rater evaluation.

Problem

• Enhancing long-context comprehension in LLMs
• Reducing hallucinations and improving transparency
• Handling complex reasoning tasks with reliability

Innovation

• Graph-based reasoning for long-context comprehension
• Directed acyclic graph construction for logical consistency
• Monte Carlo Tree Search for complex reasoning paths