When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models

2025-09-23

AI Summary

This work investigates how context length in supervised fine-tuning (SFT) affects large language models’ performance on short-context tasks. Contrary to pretraining—where extended context often degrades short-context performance—we find that long-context SFT *improves* short-context task accuracy. To explain this, we propose a decoupled analytical framework that isolates the roles of multi-head attention and feed-forward networks, revealing that long-context SFT systematically strengthens the model’s preference for *parameterized knowledge*, inducing a context-length-dependent knowledge bias; in contrast, short-context SFT favors *memory-based knowledge*. Building on this insight, we design a hybrid-context SFT strategy that preserves long-context capability while mitigating the bias. Extensive experiments validate the mechanism and demonstrate that hybrid training further boosts short-context task performance. Our findings establish a new paradigm for controllable, interpretable LLM fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) have achieved impressive performance across natural language processing (NLP) tasks. As real-world applications increasingly demand longer context windows, continued pretraining and supervised fine-tuning (SFT) on long-context data has become a common approach. While the effects of data length in continued pretraining have been extensively studied, their implications for SFT remain unclear. In this work, we systematically investigate how SFT data length influences LLM behavior on short-context tasks. Counterintuitively, we find that long-context SFT improves short-context performance, contrary to the commonly observed degradation from long-context pretraining. To uncover the underlying mechanisms of this phenomenon, we first decouple and analyze two key components, Multi-Head Attention (MHA) and Feed-Forward Network (FFN), and show that both independently benefit from long-context SFT. We further study their interaction and reveal a knowledge preference bias: long-context SFT promotes contextual knowledge, while short-context SFT favors parametric knowledge, making exclusive reliance on long-context SFT suboptimal. Finally, we demonstrate that hybrid training mitigates this bias, offering explainable guidance for fine-tuning LLMs.

Problem

• Investigates how supervised fine-tuning data length affects LLM behavior on short-context tasks
• Explores why long-context SFT improves short-context performance contrary to pretraining effects
• Analyzes knowledge preference bias between contextual and parametric knowledge in SFT

Innovation

• Long-context SFT improves short-context performance
• Analyzes MHA and FFN components independently
• Hybrid training mitigates knowledge preference bias