Dynamic Rank Adjustment for Accurate and Efficient Neural Network Training

2025-08-12

AI Summary

Low-rank training suffers from rank collapse due to rigid fixed-rank parameterization, degrading model expressivity; moreover, the effective rank of weight matrices often deteriorates progressively during optimization, exacerbating this issue. To address this, we propose a dynamic rank adjustment framework that periodically inserts full-rank training phases into low-rank optimization—thereby actively restoring the spectral richness and rank of weight matrices. Our method is built upon SVD-based parameterization and constitutes the first approach enabling alternating, synergistic optimization between low-rank and full-rank regimes. Experiments across multiple benchmark tasks demonstrate that our strategy achieves accuracy comparable to full-rank training while incurring computational overhead nearly identical to standard SVD-based low-rank training. Thus, it effectively reconciles model efficiency with representational capacity, offering a principled trade-off between parameter count, computation, and performance.

Technology Category

• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Personalized content ranking and presentation

Abstract

Low-rank training methods reduce the number of trainable parameters by re-parameterizing the weights with matrix decompositions (e.g., singular value decomposition). However, enforcing a fixed low-rank structure caps the rank of the weight matrices and can hinder the model's ability to learn complex patterns. Furthermore, the effective rank of the model's weights tends to decline during training, and this drop is accelerated when the model is reparameterized into a low-rank structure. In this study, we argue that strategically interleaving full-rank training epochs within low-rank training epochs can effectively restore the rank of the model's weights. Based on our findings, we propose a general dynamic-rank training framework that is readily applicable to a wide range of neural-network tasks. We first describe how to adjust the rank of weight matrix to alleviate the inevitable rank collapse that arises during training, and then present extensive empirical results that validate our claims and demonstrate the efficacy of the proposed framework. Our empirical study shows that the proposed method achieves almost the same computational cost as SVD-based low-rank training while achieving a comparable accuracy to full-rank training across various benchmarks.

Problem

• Addresses fixed low-rank structure limiting model learning capacity
• Mitigates rank collapse accelerated by low-rank reparameterization during training
• Balances computational efficiency with accuracy in neural network training

Innovation

• Interleaving full-rank epochs during low-rank training
• Dynamic adjustment framework preventing rank collapse
• Maintains low computational cost with full-rank accuracy