Probabilistic Iterative Hard Thresholding for Sparse Learning

2024-09-02

arXiv.org

AI Summary

For high-dimensional, small-sample learning problems involving ℓ₀-constrained expected risk minimization, existing stochastic sparse optimization methods struggle to guarantee both convergence and accurate sparse recovery under noisy gradient estimates. To address this, we propose the Probabilistic Iterative Hard Thresholding (PIHT) algorithm, which integrates stochastic optimization, expected risk minimization, and a provably falsifiable probabilistic thresholding update rule. We establish, for the first time, a rigorous convergence theory for ℓ₀-constrained stochastic processes under noisy gradients—ensuring robust convergence in large-scale, noisy-data regimes. Experiments on sparse linear regression and 1-bit compressed sensing demonstrate that PIHT significantly outperforms state-of-the-art stochastic sparse methods, achieving superior support recovery accuracy and enhanced computational stability.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

For statistical modeling wherein the data regime is unfavorable in terms of dimensionality relative to the sample size, finding hidden sparsity in the ground truth can be critical in formulating an accurate statistical model. The so-called"l0 norm"which counts the number of non-zero components in a vector, is a strong reliable mechanism of enforcing sparsity when incorporated into an optimization problem. However, in big data settings wherein noisy estimates of the gradient must be evaluated out of computational necessity, the literature is scant on methods that reliably converge. In this paper we present an approach towards solving expectation objective optimization problems with cardinality constraints. We prove convergence of the underlying stochastic process, and demonstrate the performance on two Machine Learning problems.

Problem

• Finding hidden sparsity in high-dimensional data
• Optimizing l0 norm for sparse model accuracy
• Ensuring convergence in noisy big data settings

Innovation

• Probabilistic Iterative Hard Thresholding method
• Sparse learning with l0 norm
• Convergence in noisy gradient settings