Optimal and computationally tractable lower bounds for logistic log-likelihoods

2024-10-14

AI Summary

This paper addresses the optimization difficulty of the logistic log-likelihood function in logistic regression models. We propose a novel Piecewise Quadratic Bound Lowerer (PQBL), the first analytically tractable quadratic minorizer that strictly dominates all existing tangent quadratic bounds—achieving both sharpness and computational feasibility. Methodologically, we construct an analytically characterizable piecewise quadratic minorizer and establish its explicit equivalence to a generalized Lasso problem; this minorizer is then integrated into both the Minorize-Maximize (MM) and variational Bayesian (VB) frameworks. Our main contributions are: (1) theoretically, the first unified construction of a strictly dominating tangent-bound minorizer; and (2) empirically, substantial acceleration of MM convergence for penalized maximum likelihood estimation and improved VB inference accuracy—while preserving closed-form updates and low computational overhead.

Technology Category

• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

The logit transform is arguably the most widely-employed link function beyond linear settings. This transformation routinely appears in regression models for binary data and provides a central building-block in popular methods for both classification and regression. Its widespread use, combined with the lack of analytical solutions for the optimization of objective functions involving the logit transform, still motivates active research in computational statistics. Among the directions explored, a central one has focused on the design of tangent lower bounds for logistic log-likelihoods that can be tractably optimized, while providing a tight approximation of these log-likelihoods. This has led to the development of effective minorize-maximize (MM) algorithms for point estimation, and variational schemes for approximate Bayesian inference under several logit models. However, the overarching focus has been on tangent quadratic minorizers. In fact, it is still unclear whether tangent lower bounds sharper than quadratic ones can be derived without undermining the tractability of the resulting minorizer. This article addresses such a question through the design and study of a novel piece-wise quadratic lower bound that uniformly improves any tangent quadratic minorizer, including the sharpest ones, while admitting a direct interpretation in terms of the classical generalized lasso problem. As illustrated in realistic empirical studies, such a sharper bound not only improves the speed of convergence of common MM schemes for penalized maximum likelihood estimation, but also yields tractable variational Bayes (VB) approximations with higher accuracy relative to those obtained under popular quadratic bounds employed in VB.

Problem

• Developing sharper piece-wise quadratic bounds for logistic log-likelihoods optimization
• Improving convergence speed of MM algorithms for penalized maximum likelihood estimation
• Enhancing variational Bayes accuracy through tractable non-quadratic minorizers

Innovation

• Piece-wise quadratic lower bound for logistic log-likelihoods
• Uniform improvement over tangent quadratic minorizers
• Interpretable as generalized lasso problem for tractability