Sparsity via Hyperpriors: A Theoretical and Algorithmic Study under Empirical Bayes Framework

2025-11-09

AI Summary

This work investigates the mechanistic impact of hyperprior selection in the empirical Bayes framework (EBF) on sparsity, local optimality, stability, and noise robustness of sparse learning solutions. Theoretically, it establishes an intrinsic link between hyperprior monotonicity—e.g., semilaplace-type priors—and enhanced solution sparsity as well as reduced numbers of local minima. It further derives a quantitative model characterizing how noise level and inverse problem ill-conditioning jointly affect solution bias and convergence behavior in EBF. Methodologically, we design a proximal alternating linearized minimization (PALM) algorithm with guaranteed global convergence, capable of unifying optimization for both convex and nonconvex hyperpriors. Evaluated on 2D image deblurring, our approach achieves significant improvements: average sparsity increase of 23.6% and mean PSNR gain of 4.1 dB, while maintaining robustness under high noise and severe ill-conditioning.

Technology Category

• Machine Learning: Bayesian Learning
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

This paper presents a comprehensive analysis of hyperparameter estimation within the empirical Bayes framework (EBF) for sparse learning. By studying the influence of hyperpriors on the solution of EBF, we establish a theoretical connection between the choice of the hyperprior and the sparsity as well as the local optimality of the resulting solutions. We show that some strictly increasing hyperpriors, such as half-Laplace and half-generalized Gaussian with the power in $(0,1)$, effectively promote sparsity and improve solution stability with respect to measurement noise. Based on this analysis, we adopt a proximal alternating linearized minimization (PALM) algorithm with convergence guaranties for both convex and concave hyperpriors. Extensive numerical tests on two-dimensional image deblurring problems demonstrate that introducing appropriate hyperpriors significantly promotes the sparsity of the solution and enhances restoration accuracy. Furthermore, we illustrate the influence of the noise level and the ill-posedness of inverse problems to EBF solutions.

Problem

• Estimate hyperparameters in empirical Bayes framework for sparse learning
• Analyze hyperprior influence on sparsity and solution optimality
• Develop algorithms for image deblurring with enhanced restoration accuracy

Innovation

• Hyperpriors promote sparsity and stability
• Proximal alternating linearized minimization algorithm used
• Applied to image deblurring with enhanced accuracy