Learning a Gaussian Mixture for Sparsity Regularization in Inverse Problems

2024-01-29

IMA Journal of Numerical Analysis

AI Summary

Modeling sparse priors under generic bases remains challenging for linear inverse problems. Method: This paper proposes a probabilistic sparse prior based on a Degenerate Gaussian Mixture Model (DGMM), integrated into a neural network architecture that ensures both Bayesian interpretability and end-to-end differentiability. For the first time, DGMM is embedded explicitly to realize Maximum A Posteriori (MAP) estimation, enabling joint supervised and unsupervised training. Results: Evaluated on multiple 1D inverse problems, the method consistently outperforms LASSO, group LASSO, iterative hard thresholding, and sparse coding in reconstruction mean squared error, while demonstrating strong robustness to noise and model mismatch. Key contributions include: (i) the first DGMM-based sparse modeling framework for generic bases; and (ii) the first differentiable neural integration of DGMM grounded in Bayesian semantics.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In inverse problems it is widely recognized that the incorporation of a sparsity prior yields a regularization effect on the solution. This approach is grounded on the a priori assumption that the unknown can be appropriately represented in a basis with a limited number of significant components while most coefficients are close to zero. This occurrence is frequently observed in real-world scenarios, such as with piecewise smooth signals. In this study we propose a probabilistic sparsity prior formulated as a mixture of degenerate Gaussians, capable of modelling sparsity with respect to a generic basis. Under this premise we design a neural network that can be interpreted as the Bayes estimator for linear inverse problems. Additionally, we put forth both a supervised and an unsupervised training strategy to estimate the parameters of this network. To evaluate the effectiveness of our approach we conduct a numerical comparison with commonly employed sparsity-promoting regularization techniques, namely Least Absolute Shrinkage and Selection Operator (LASSO), group LASSO, iterative hard thresholding and sparse coding/dictionary learning. Notably, our reconstructions consistently exhibit lower mean square error values across all one-dimensional datasets utilized for the comparisons, even in cases where the datasets significantly deviate from a Gaussian mixture model.

Problem

• Modeling sparsity with Gaussian mixture for inverse problems
• Designing neural network as Bayes estimator for linear inverse problems
• Comparing effectiveness with LASSO and other sparsity techniques

Innovation

• Mixture of degenerate Gaussians for sparsity prior
• Neural network as Bayes estimator
• Supervised and unsupervised training strategies