🤖 AI Summary
This work investigates efficient adaptation of general-purpose large language models to specialized domains—such as mathematics, instruction following, and code—through continual pretraining, balancing performance and computational efficiency. Spectral analysis via singular value decomposition reveals that adaptation is primarily driven by the evolution of singular vectors, while the singular value spectrum remains remarkably invariant. Leveraging the heterogeneity among attention heads, the authors propose an importance criterion to selectively roll back low-importance heads, reducing up to 60% of updated parameters without compromising model quality and even improving benchmark accuracy by up to 4%. The study further uncovers linearly interpolable domain connectivity across distinct domains and releases Diffract, a large-scale spectral analysis toolkit.
📝 Abstract
We study continual pre-training (CPT) as a mechanism for adapting general-purpose large language models to specialized domains: mathematics, instruction, code, and natural text. Using singular value decomposition of weight matrices, we find that CPT leaves singular value spectra largely invariant, with adaptation driven mainly by changes in singular vectors. An analysis of attention-head projection matrices reveals strong, domain-dependent head heterogeneity, which we exploit to define a head importance criterion: up to 60% of head updates can be removed without measurable quality loss. Selectively rewinding low-importance heads to their pre-trained state improves benchmark accuracy by up to 4% versus the fully trained baseline. Finally, we identify domain connectivity - linear interpolation between CPT checkpoints yields smooth domain-quality interpolation without notable degradation on either domain - and release Diffract, an open-source toolkit for scalable spectral analysis of billion-parameter models.