Empirical Bayes Estimation for Lasso-Type Regularizers: Analysis of Automatic Relevance Determination

2025-01-20

AI Summary

This work establishes the theoretical foundation for automatic relevance determination (ARD) in linear regression models with non-conjugate sparse regularizers—including Lasso and Group Lasso—under empirical Bayes estimation. We conduct rigorous asymptotic analysis combined with sparse regularization theory to prove, for the first time, that empirical Bayes can spontaneously induce ARD even without conjugate priors. We precisely characterize the necessary and sufficient conditions for ARD emergence: they depend on the local coherence of the design matrix and the noise level. Our analysis unifies the understanding of ARD capabilities across Ridge, Lasso, and Group Lasso, revealing a fundamental tension between sparsity and adaptivity. This provides the first general theoretical guarantee for adaptive sparse learning, bridging interpretability and practicality in Bayesian sparse modeling.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

This paper focuses on linear regression models with non-conjugate sparsity-inducing regularizers such as lasso and group lasso. Although empirical Bayes approach enables us to estimate the regularization parameter, little is known on the properties of the estimators. In particular, there are many unexplained aspects regarding the specific conditions under which the mechanism of automatic relevance determination (ARD) occurs. In this paper, we derive the empirical Bayes estimators for the group lasso regularized linear regression models with a limited number of parameters. It is shown that the estimators diverge under a certain condition, giving rise to the ARD mechanism. We also prove that empirical Bayes methods can produce ARD mechanism in general regularized linear regression models and clarify the conditions under which models such as ridge, lasso, and group lasso can produce ARD mechanism.

Problem

• Automatic Relevance Determination
• Lasso Regression
• Ridge Regression

Innovation

• Bayesian Methods
• Automatic Relevance Determination (ARD)
• Empirical Bayes Estimation