Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning

2025-09-03

AI Summary

This work investigates the intrinsic mechanisms by which pretraining enhances downstream classification performance in NLP. Using a BERT-6 architecture pretrained on Wikipedia and fine-tuned on FewRel and DBpedia, we analyze token-level confusion matrices and representation dynamics across layers. We find that pretraining progressively strengthens high-level semantic modeling by breaking token symmetry and inducing robust semantic clustering layerwise. We propose the Alignment-Perception Threshold (APT) as an order parameter to quantitatively characterize pretraining efficacy; its evolutionary trajectory aligns with findings in vision, suggesting a cross-modal universal mechanism. Experiments demonstrate that pretraining substantially improves fine-tuning accuracy, with output-layer representations vastly outperforming embedding-layer ones, and label prediction confidence remaining stable—confirming effective capture of abstract linguistic structure.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Natural language processing (NLP) enables the understanding and generation of meaningful human language, typically using a pre-trained complex architecture on a large dataset to learn the language and next fine-tune its weights to implement a specific task. Twofold goals are examined; to understand the mechanism underlying successful pre-training and to determine the interplay between the pre-training accuracy and the fine-tuning of classification tasks. The following main results were obtained; the accuracy per token (APT) increased with its appearance frequency in the dataset, and its average over all tokens served as an order parameter to quantify pre-training success, which increased along the transformer blocks. Pre-training broke the symmetry among tokens and grouped them into finite, small, strong match token clusters, as inferred from the presented token confusion matrix. This feature was sharpened along the transformer blocks toward the output layer, enhancing its performance considerably compared with that of the embedding layer. Consequently, higher-order language structures were generated by pre-training, even though the learning cost function was directed solely at identifying a single token. These pre-training findings were reflected by the improved fine-tuning accuracy along the transformer blocks. Additionally, the output label prediction confidence was found to be independent of the average input APT, as the input meaning was preserved since the tokens are replaced primarily by strong match tokens. Finally, although pre-training is commonly absent in image classification tasks, its underlying mechanism is similar to that used in fine-tuning NLP classification tasks, hinting at its universality. The results were based on the BERT-6 architecture pre-trained on the Wikipedia dataset and fine-tuned on the FewRel and DBpedia classification tasks.

Problem

• Understanding learning mechanisms in NLP pre-training and fine-tuning
• Examining interplay between pre-training accuracy and fine-tuning performance
• Investigating token clustering and symmetry breaking in transformer blocks

Innovation

• Pre-training increases token accuracy with frequency
• Token clustering sharpens along transformer blocks
• Fine-tuning accuracy improves with pre-training success