Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning

2025-02-21

AI Summary

This work addresses the high cost and poor generalizability of instruction data construction in post-training long-context large language models (LLMs). We propose an efficient instruction-tuning data synthesis framework that leverages pretrained LLMs for controllable, automatic context expansion—combined with instruction-difficulty-aware sampling, structured background injection, and long-document-level instruction-pair synthesis—to generate high-fidelity, long-context instruction data at low cost. Key findings reveal that short-context instruction data exhibits cross-length generalization capability, and both instruction difficulty and context composition significantly influence fine-tuning performance. On the LongBench document-level benchmark, our synthesized data achieves performance comparable to human-annotated data and substantially outperforms existing synthetic approaches. This work establishes a novel paradigm for efficient post-training of long-context LLMs.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Long-context modelling for large language models (LLMs) has been a key area of recent research because many real world use cases require reasoning over longer inputs such as documents. The focus of research into modelling long context has been on how to model position and there has been little investigation into other important aspects of language modelling such as instruction tuning. Long context training examples are challenging and expensive to create and use. In this paper, we investigate how to design instruction data for the post-training phase of a long context pre-trained model: how much and what type of context is needed for optimal and efficient post-training. Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones, while also identifying other critical factors such as instruction difficulty and context composition. Based on these findings, we propose context synthesis, a novel data synthesis framework that leverages off-the-shelf LLMs to generate extended background contexts for high-quality instruction-answer pairs. Experiment results on the document-level benchmark (LongBench) demonstrate that our proposed approach outperforms previous instruction synthesis approaches and comes close to the performance of human-annotated long-context instruction data. The project will be available at: https://github.com/NJUNLP/context-synthesis.

Problem

• Optimizing instruction data for long-context LLMs.
• Reducing cost of creating long-context training examples.
• Improving generalization from short to long contexts.

Innovation

• Leverages LLMs for context synthesis
• Focuses on instruction tuning efficiency
• Generalizes short-context models to long-context