SLORR: Simple and Efficient In-Training Low-Rank Regularization

2026-07-09

AI Summary

This work addresses the challenge of efficiently achieving low-rank compression during neural network training, a task hindered by existing methods that often incur high computational overhead, require architectural modifications, or rely on stateful caching mechanisms. The authors propose SLORR, a novel framework that, for the first time, enables stateless, architecture-preserving low-rank regularization directly during training. SLORR imposes regularizers—based on either Hoyer sparsity or the nuclear norm—on weight matrices without introducing auxiliary parameters or caches, and employs GPU-friendly approximations for both forward and backward passes, accompanied by theoretical guarantees. Experiments demonstrate that SLORR substantially improves the performance of compressed models on ImageNet classification and large language model pretraining, with only marginal increases in training cost—under 8% for vision tasks and merely 1% for language tasks.

Abstract

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.

Problem

• low-rank factorization
• neural network compression
• in-training regularization
• accuracy loss
• model compressibility

Innovation

• low-rank regularization
• in-training compression
• GPU-friendly approximation
• architecture-preserving
• Hoyer sparsity