An Efficient Newton Algorithm for Nonnegative Matrix Factorization with the Kullback-Leibler Divergence

2026-07-15

AI Summary

This work addresses the inefficiency and weak convergence often encountered in Kullback–Leibler divergence-based non-negative matrix factorization (KL-NMF) when applied to Poisson count data. To overcome these limitations, we propose a novel Newton-type algorithm that constructs a non-separable surrogate function based on the second-order Taylor expansion of the loss function and solves it efficiently via a generalized hierarchical alternating least squares (HALS) strategy. By incorporating second-order information into the KL-NMF optimization framework for the first time, our method transcends the performance bottleneck of conventional first-order separable surrogate approaches, achieving both guaranteed convergence and significantly improved computational efficiency. Experimental results on multiple real-world datasets demonstrate that the proposed algorithm consistently outperforms state-of-the-art methods, exhibiting faster convergence and higher computational throughput.

Abstract

Nonnegative Matrix Factorization (NMF) is a fundamental tool in unsupervised learning, which approximates a nonnegative matrix by the product of two low-rank nonnegative factors. The Kullback-Leibler (KL) divergence is best suited to measure the data to model discrepancy when the decomposed data sample follows a Poisson distribution, which is the case for count datasets such as term-document matrices or images. Most KL-NMF algorithms in the literature minimize a separable majorant of the loss to find their next iterate. We argue that this method has reached its limits and propose to use instead the second-order Taylor expansion of the loss, leading to a Newton-type method. We minimize this non-separable surrogate by proposing a generalization of the well-known HALS algorithm. This yields an efficient KL-NMF algorithm which provably converges and which competes favorably with state-of-the-art algorithms on a large variety of datasets.

Problem

• Nonnegative Matrix Factorization
• Kullback-Leibler Divergence
• Newton Algorithm
• Poisson Distribution
• Unsupervised Learning

Innovation

• Newton-type algorithm
• Kullback-Leibler divergence
• Nonnegative Matrix Factorization
• second-order Taylor expansion
• HALS generalization