Bayesian Variable Selection in Generalized Linear Models

2026-06-23

AI Summary

This work proposes the first fully Bayesian hierarchical approach for covariate selection in generalized linear models that simultaneously achieves full conjugacy, posterior consistency, and broad applicability across exponential family distributions. By introducing binary inclusion indicators to explicitly model whether each covariate enters the linear predictor, the method unifies variable selection and parameter estimation within a single coherent framework, effectively accounting for model uncertainty. Built upon conjugate priors, the approach enables efficient Gibbs sampling and is accompanied by an R package for practical implementation. Theoretical analysis establishes posterior consistency for both the inclusion indicators and the active regression coefficients. Extensive experiments on synthetic and real-world datasets demonstrate superior performance in terms of predictive accuracy and statistical inference.

Abstract

Covariate selection in Generalized Linear Models (GLMs) is a fundamental problem in statistics, as including irrelevant predictors might lead to overfitting and poor interpretability, while omitting relevant ones might result in biased estimates. Most Bayesian approaches to variable selection -- including spike-and-slab priors and continuous shrinkage priors -- have key limitations, e.g., (i) are based on non fully conjugate formulations, (ii) are restricted to a linear model, or (iii) lack posterior consistency guarantees for the variable selection procedure and model parameters. In this work, we propose a fully Bayesian hierarchical and conjugate framework for covariate selection in GLMs, applicable to any distribution in the exponential family, based on modeling a binary inclusion indicator that directly encodes covariate inclusion in the linear predictor. In our approach, variable selection and parameter estimation are performed simultaneously, incorporating both sources of uncertainty in posterior inference. Consequently, our methodology provides a valid post-model Bayesian selection procedure. We present theoretical guarantees of the proposed fully conjugate Bayesian variable selection for GLMs, establishing posterior consistency of both the inclusion indicators and the active regression coefficients. We derive an efficient Gibbs Sampling algorithm with a corresponding R package implementation. We validate the proposed method on synthetic and real-world datasets, demonstrating competitive predictive and inferential performance.

Problem

• Bayesian variable selection
• Generalized Linear Models
• covariate selection
• posterior consistency
• spike-and-slab priors

Innovation

• Bayesian variable selection
• Generalized Linear Models
• posterior consistency
• conjugate priors
• Gibbs sampling