DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

2026-06-27

AI Summary

This work proposes DLR, a training-only, parameter-free, and losslessly foldable plugin that enhances the representational capacity of low-rank pretraining by injecting fixed structured residuals into low-rank outputs. Unlike conventional low-rank approaches that often compromise model performance to reduce training costs, DLR achieves the first low-rank training enhancement with zero inference overhead and no additional parameters, while supporting closed-form weight fusion to fully preserve deployment efficiency. Evaluated on LLaMA models ranging from 60M to 7B parameters, DLR consistently improves perplexity on the C4 validation set—particularly for models exceeding 130M parameters—and seamlessly transfers to downstream supervised fine-tuning tasks without modification.

Abstract

Large language models have driven recent progress in language and multimodal AI, yet pre-training them at scale is prohibitively expensive. Low-rank pre-training, which factorizes each weight matrix into a rank-r product to reduce both parameters and FLOPs, is a promising response but typically lags full-rank training in quality. We propose Duplicated Latent Residual (DLR), a training-only, parameter-free, foldable plug-in for low-rank pre-training. DLR augments the standard low-rank output Bz with a fixed structured residual alpha/sqrt(K) * Expand_K(z) that replicates each latent coordinate K = ceil(d_out/r) times across the output. With alpha fixed, DLR adds zero learnable parameters per layer; after training, it is absorbed into the up-projection in closed form, B* = B + alpha/sqrt(K) R^T, so deployment parameter count, FLOPs and memory match the underlying low-rank backbone exactly. Across LLaMA models from 60M to 7B parameters, DLR strengthens low-rank pre-training on C4 validation perplexity in most settings, with the clearest gains at 130M and above; folded checkpoints transfer cleanly to supervised fine-tuning on standard benchmarks.

Problem

• low-rank pre-training
• large language models
• training cost
• model quality
• parameter efficiency

Innovation

• low-rank pre-training
• zero-inference-cost
• latent residual
• parameter-free plug-in
• foldable architecture