Efficient Algorithms for Regularized Nonnegative Scale-invariant Low-rank Approximation Models

2024-03-27

arXiv.org

AI Summary

To address the challenges of difficult regularization parameter tuning, slow convergence, and poor interpretability in sparse nonnegative matrix/tensor decomposition, this paper proposes a homogeneous scale-invariant low-rank approximation model. We rigorously prove for the first time that scale invariance induces an implicit ℓₚ norm regularization effect, thereby enhancing solution balance and interpretability. Building upon this insight, we develop a general Majorization-Minimization (MM) optimization framework that unifies support for non-Euclidean loss functions and explicit ℓₚ regularization, while providing theoretical convergence guarantees and intrinsic hyperparameter self-guidance. Extensive experiments on sparse NMF, ridge-regularized CP decomposition, and sparse Tucker decomposition demonstrate that our method significantly accelerates convergence (average speedup of 2.1×), improves model stability, and enhances both factor sparsity and semantic interpretability.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Regularized nonnegative low-rank approximations, such as sparse Nonnegative Matrix Factorization or sparse Nonnegative Tucker Decomposition, form an important branch of dimensionality reduction models known for their enhanced interpretability. From a practical perspective, however, selecting appropriate regularizers and regularization coefficients, as well as designing efficient algorithms, remains challenging due to the multifactor nature of these models and the limited theoretical guidance available. This paper addresses these challenges by studying a more general model, the Homogeneous Regularized Scale-Invariant model. We prove that the scale-invariance inherent to low-rank approximation models induces an implicit regularization effect that balances solutions. This insight provides a deeper understanding of the role of regularization functions in low-rank approximation models, informs the selection of regularization hyperparameters, and enables the design of balancing strategies to accelerate the empirical convergence of optimization algorithms. Additionally, we propose a generic Majorization-Minimization (MM) algorithm capable of handling $ell_p^p$-regularized nonnegative low-rank approximations with non-Euclidean loss functions, with convergence guarantees. Our contributions are demonstrated on sparse Nonnegative Matrix Factorization, ridge-regularized Nonnegative Canonical Polyadic Decomposition, and sparse Nonnegative Tucker Decomposition.

Problem

• Sparse Non-negative Matrix/Tucker Decomposition
• Dimensionality Reduction
• Algorithm Optimization

Innovation

• Homogeneous Regularization Scale Invariant Model
• Majorization-Minimization Algorithm
• Automatic Regularization Parameter Adjustment