Variational Garrote for Statistical Physics-based Sparse and Robust Variable Selection

2025-09-08

AI Summary

In high-dimensional sparse regression, conventional methods (e.g., LASSO, Ridge) suffer from inconsistent variable selection, sharp degradation in generalization performance under high sparsity, and inability to accurately estimate the true number of relevant features. Method: We propose the Variational Garrote Enhanced Model, which introduces explicit binary spin variables to explicitly model feature selection and leverages statistical physics insights to reveal a phase transition in generalization performance induced by overparameterization. This phase transition point serves as a robust signal for precise estimation of the true sparsity level. The method employs variational inference to construct a differentiable loss function and utilizes automatic differentiation for efficient optimization. Contribution/Results: Experiments on synthetic and real-world datasets demonstrate that our approach significantly outperforms LASSO and Ridge—achieving higher variable selection consistency, greater stability, and lower generalization error—while remaining applicable to compressive sensing and model pruning tasks.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Selecting key variables from high-dimensional data is increasingly important in the era of big data. Sparse regression serves as a powerful tool for this purpose by promoting model simplicity and explainability. In this work, we revisit a valuable yet underutilized method, the statistical physics-based Variational Garrote (VG), which introduces explicit feature selection spin variables and leverages variational inference to derive a tractable loss function. We enhance VG by incorporating modern automatic differentiation techniques, enabling scalable and efficient optimization. We evaluate VG on both fully controllable synthetic datasets and complex real-world datasets. Our results demonstrate that VG performs especially well in highly sparse regimes, offering more consistent and robust variable selection than Ridge and LASSO regression across varying levels of sparsity. We also uncover a sharp transition: as superfluous variables are admitted, generalization degrades abruptly and the uncertainty of the selection variables increases. This transition point provides a practical signal for estimating the correct number of relevant variables, an insight we successfully apply to identify key predictors in real-world data. We expect that VG offers strong potential for sparse modeling across a wide range of applications, including compressed sensing and model pruning in machine learning.

Problem

• Sparse variable selection from high-dimensional data
• Robust feature selection under varying sparsity levels
• Identifying correct number of relevant variables

Innovation

• Variational Garrote with spin variables
• Automatic differentiation for scalable optimization
• Robust selection via uncertainty transition analysis