Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models

2026-04-11

AI Summary

This study addresses the prevalent yet underexplored phenomenon of Incomplete Learning in Supervised Fine-Tuning (ILP)—where models fail to reproduce certain training samples even after convergence. The work formally defines ILP for the first time and introduces a diagnostic framework based on observable signals to identify five heterogeneous root causes. Targeted intervention strategies are then proposed to mitigate each cause. Through controlled cross-dataset and cross-domain experiments coupled with causal analysis on models including Qwen, LLaMA, and OLMo2, the authors demonstrate the ubiquity and complexity of ILP, revealing that aggregate performance gains often obscure localized learning failures. The findings underscore the necessity of fine-grained diagnosis of fine-tuning deficiencies and offer actionable pathways to enhance the reliability of supervised fine-tuning.

Technology Category

• Machine Learning: Causal Learning
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Supervised Fine-Tuning (SFT) is the standard approach for adapting large language models (LLMs) to downstream tasks. However, we observe a persistent failure mode: even after convergence, models often fail to correctly reproduce a subset of their own supervised training data. We refer to this behavior as the Incomplete Learning Phenomenon(ILP). This paper presents the first systematic study of ILP in LLM fine-tuning. We formalize ILP as post-training failure to internalize supervised instances and demonstrate its prevalence across multiple model families, domains, and datasets. Through controlled analyses, we identify five recurrent sources of incomplete learning: (1) missing prerequisite knowledge in the pre-trained model, (2) conflicts between SFT supervision and pre-training knowledge, (3) internal inconsistencies within SFT data, (4) left-side forgetting during sequential fine-tuning, and (5) insufficient optimization for rare or complex patterns. We introduce a diagnostic-first framework that maps unlearned samples to these causes using observable training and inference signals, and study several targeted mitigation strategies as causal interventions. Experiments on Qwen, LLaMA, and OLMo2 show that incomplete learning is widespread and heterogeneous, and that improvements in aggregate metrics can mask persistent unlearned subsets. The findings highlight the need for fine-grained diagnosis of what supervised fine-tuning fails to learn, and why.

Problem

• Supervised Fine-Tuning
• Large Language Models
• Incomplete Learning
• Training Failure
• Model Adaptation

Innovation

• Incomplete Learning Phenomenon
• Supervised Fine-Tuning
• Diagnostic Framework
• Causal Intervention
• Large Language Models