🤖 AI Summary
This work addresses the lack of effective, interpretable, and low-cost guidance for determining layer-wise freezing or training strategies during continual pre-training of large language models. The authors propose LayerTracer, a framework that identifies task-executing layers, quantifies layer sensitivity, and analyzes the stability of representational evolution across layers. Their analysis reveals that deeper layers are primarily responsible for task execution and exhibit greater robustness to perturbations. Based on these insights, they introduce an interpretable and cost-efficient continual pre-training paradigm—freezing deeper layers while fine-tuning shallower ones—and extend it to hybrid model construction. The approach is architecture-agnostic and, through representation analysis and controlled experiments, consistently outperforms full-parameter fine-tuning and alternative strategies on C-Eval and CMMLU benchmarks, while demonstrating that embedding high-quality deep modules effectively preserves original knowledge.
📝 Abstract
Selective layer-wise updates are essential for low-cost continued pre-training of Large Language Models (LLMs), yet determining which layers to freeze or train remains an empirical black-box problem due to the lack of interpretable guidance. To address this issue, we propose LayerTracer, an architecture-agnostic diagnostic framework that reveals the evolution patterns of layer-wise representations and stability by locating task execution positions and quantifying layer sensitivity. Analysis results reveal that deep layers act as critical regions for task execution and maintain high stability against disruptive updates. Guided by this finding, we conduct three controlled continued pre-training trials to compare diverse freeze-train strategies, demonstrating that training shallow layers while freezing deep layers consistently outperforms full-parameter fine-tuning and the opposite allocation on both C-Eval and CMMLU benchmarks. We further present a hybrid model case study, which validates that placing high-quality pre-trained modules in deep layers effectively preserves inherent knowledge of the model. This work delivers a low-cost and interpretable solution for resource-constrained teams, offering actionable guidance for layer-wise parameter allocation in continued pre-training and hybrid model construction.