The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuning

2026-02-11

AI Summary

This study investigates the mechanisms of capability transfer in language models from pretraining to supervised fine-tuning and the factors influencing its reliability. By constructing a multidimensional correlation analysis framework, the authors systematically evaluate how different data mixing strategies and model scales affect cross-stage alignment between pretraining and fine-tuning in terms of both accuracy and confidence. The work presents the first quantitative evidence that the stability of this transfer varies significantly: accuracy and confidence exhibit distinct—and sometimes opposing—scaling behaviors. Moreover, the study identifies cross-stage reliable prediction benchmarks and reveals substantial heterogeneity in transfer reliability across capability types, benchmark tasks, and model scales, offering empirical guidance for efficient model development, data curation, and benchmark selection.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Understanding how language model capabilities transfer from pretraining to supervised fine-tuning (SFT) is fundamental to efficient model development and data curation. In this work, we investigate four core questions: RQ1. To what extent do accuracy and confidence rankings established during pretraining persist after SFT? RQ2. Which benchmarks serve as robust cross-stage predictors and which are unreliable? RQ3. How do transfer dynamics shift with model scale? RQ4. How well does model confidence align with accuracy, as a measure of calibration quality? Does this alignment pattern transfer across training stages? We address these questions through a suite of correlation protocols applied to accuracy and confidence metrics across diverse data mixtures and model scales. Our experiments reveal that transfer reliability varies dramatically across capability categories, benchmarks, and scales -- with accuracy and confidence exhibiting distinct, sometimes opposing, scaling dynamics. These findings shed light on the complex interplay between pretraining decisions and downstream outcomes, providing actionable guidance for benchmark selection, data curation, and efficient model development.

Problem

• knowledge transfer
• pretraining
• supervised fine-tuning
• model calibration
• transfer dynamics

Innovation

• knowledge transfer
• pretraining
• supervised fine-tuning
• model calibration
• scaling dynamics