Alternating Direction Method of Multipliers for Nonlinear Matrix Decompositions

2025-12-19

AI Summary

This paper addresses the nonlinear matrix decomposition (NMD) problem: given an input matrix $X$ and target rank $r$, find low-rank factors $W$ and $H$ such that $X approx f(WH)$, where $f$ is an element-wise nonlinearity. We systematically introduce the alternating direction method of multipliers (ADMM) into the NMD framework—enabling support for arbitrary differentiable or nondifferentiable nonlinearities (e.g., ReLU, square, MinMax) and composite loss functions (e.g., least squares, $ell_1$, KL divergence). Theoretically, we establish convergence guarantees for the proposed algorithm under nonconvex settings. Empirically, our method achieves significant improvements in accuracy and generalization across diverse tasks—including sparse nonnegative data fitting, probabilistic circuit modeling, and recommendation systems—while maintaining computational scalability and numerical stability.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present an algorithm based on the alternating direction method of multipliers (ADMM) for solving nonlinear matrix decompositions (NMD). Given an input matrix $X in mathbb{R}^{m imes n}$ and a factorization rank $r ll min(m, n)$, NMD seeks matrices $W in mathbb{R}^{m imes r}$ and $H in mathbb{R}^{r imes n}$ such that $X approx f(WH)$, where $f$ is an element-wise nonlinear function. We evaluate our method on several representative nonlinear models: the rectified linear unit activation $f(x) = max(0, x)$, suitable for nonnegative sparse data approximation, the component-wise square $f(x) = x^2$, applicable to probabilistic circuit representation, and the MinMax transform $f(x) = min(b, max(a, x))$, relevant for recommender systems. The proposed framework flexibly supports diverse loss functions, including least squares, $ell_1$ norm, and the Kullback-Leibler divergence, and can be readily extended to other nonlinearities and metrics. We illustrate the applicability, efficiency, and adaptability of the approach on real-world datasets, highlighting its potential for a broad range of applications.

Problem

• Proposes ADMM algorithm for nonlinear matrix decomposition
• Solves approximation X ≈ f(WH) with element-wise nonlinear function
• Supports diverse loss functions for real-world applications

Innovation

• ADMM algorithm for nonlinear matrix decomposition
• Supports diverse loss functions and nonlinearities
• Evaluated on real-world datasets for broad applications