LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

2026-07-07

AI Summary

Existing approaches to synthesizing supervised fine-tuning data for long-context language models suffer from three key limitations: narrow task coverage, insufficient instruction difficulty, and poor reasoning fidelity. To address these issues, this work proposes LongCrafter, a novel framework that introduces a hierarchical taxonomy encompassing 32 fine-grained long-context tasks and leverages evidence graphs to model inter-paragraph dependencies. This enables task-aligned long-text generation and the synthesis of instruction–response pairs with traceable reasoning chains and controllable difficulty levels. Experimental results demonstrate that LongCrafter significantly improves model performance on LongBench, LongBench v2, and LooGLE benchmarks, outperforming existing SFT baselines and official post-trained models—particularly on high-difficulty tasks—and effectively mitigates the “lost-in-the-middle” phenomenon.

Abstract

Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision. We propose \textbf{LongCrafter}, a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline. The taxonomy organizes long-context understanding into local/shallow and global/deep levels and yields 32 fine-grained task types that serve as a global generative prior. Guided by this taxonomy, LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans, ensuring both controllable difficulty and faithful, traceable reasoning. Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models on LongBench, LongBench~v2, and LooGLE across both Qwen2.5-7B and LLaMA-3.1-8B, with the largest gains on high-difficulty tasks. Further analysis shows that LongCrafter data is more diverse and better spread across difficulty levels, and that the trained models locate evidence robustly regardless of position, effectively mitigating the ``lost in the middle'' problem.

Problem

• long-context understanding
• supervised fine-tuning
• instruction synthesis
• faithfulness supervision
• task diversity

Innovation

• long-context understanding
• instruction synthesis
• evidence graph
• supervised fine-tuning
• task taxonomy