Efficient Pre-Training of LLMs through Truncated SVD Layers

2026-05-27

AI Summary

This work addresses the high computational cost of large language model (LLM) pretraining due to massive parameter counts, a challenge exacerbated by the inability of existing low-rank methods to simultaneously support dynamic rank selection and strict weight orthogonality. The authors propose the TSVD framework, which maintains a low-rank structure with orthogonality constraints throughout pretraining. It employs a spectral energy–based heuristic to adaptively determine the optimal rank and introduces a caching mechanism to efficiently preserve orthogonality. TSVD is the first method to concurrently enable dynamic low-rank adaptation and rigorous orthogonal constraints during training. Experiments demonstrate that it matches or exceeds full-parameter baseline performance across multiple model scales while substantially reducing computational overhead, establishing a new paradigm for efficient and scalable LLM pretraining.

Abstract

The massive scaling of Large Language Models (LLMs) has made pretraining increasingly cost-prohibitive. While low-rank representation and orthonormal weight matrices could in principle reduce parameter counts and computational overhead, most existing methods rely on static rank selection and do not enforce weight orthonormality due to high computational cost. This paper introduces TSVD, a framework that maintains low rank and strict orthonormality throughout the training process. It utilizes a spectral energy-based heuristic for adaptive rank selection, and a caching mechanisms to maintain orthonormality. Theoretical analysis justifies the advantage of the approach in pretraining dynamics and experiments across various model scales demonstrate that it is effective empirically. TSVD matches or exceeds the performance of full-parameter baselines while significantly reducing compute requirements. The approach thus offers a well-founded, practical, and scalable path toward efficient high-performance LLM pretraining.

Problem

• Large Language Models
• pre-training
• low-rank
• orthonormality
• computational cost

Innovation

• Truncated SVD
• low-rank representation
• orthonormal weight matrices
• adaptive rank selection
• efficient pretraining