Proximal Operators of Sorted Nonconvex Penalties

2025-06-18

AI Summary

This work addresses automatic variable grouping in sparse signal recovery, proposing a class of generalized sorted nonconvex penalties—including sorted MCP, SCAD, and ℓ_q—that jointly enforce clustering consistency and control coefficient estimation bias. Methodologically, we establish, for the first time, a systematic theoretical framework for the proximal operators of sorted nonconvex penalties; further, we adapt the Pool Adjacent Violators (PAV) algorithm to efficiently solve the associated nonconvex proximal problems and provide a structural characterization of their solutions. Under generalized linear models, extensive experiments demonstrate that the proposed approach significantly outperforms existing methods in variable selection accuracy, grouping consistency, and prediction performance, while achieving millisecond-level proximal computation time. The method thus achieves a favorable balance between statistical efficacy and computational efficiency.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This work studies the problem of sparse signal recovery with automatic grouping of variables. To this end, we investigate sorted nonsmooth penalties as a regularization approach for generalized linear models. We focus on a family of sorted nonconvex penalties which generalizes the Sorted L1 Norm (SLOPE). These penalties are designed to promote clustering of variables due to their sorted nature, while the nonconvexity reduces the shrinkage of coefficients. Our goal is to provide efficient ways to compute their proximal operator, enabling the use of popular proximal algorithms to solve composite optimization problems with this choice of sorted penalties. We distinguish between two classes of problems: the weakly convex case where computing the proximal operator remains a convex problem, and the nonconvex case where computing the proximal operator becomes a challenging nonconvex combinatorial problem. For the weakly convex case (e.g. sorted MCP and SCAD), we explain how the Pool Adjacent Violators (PAV) algorithm can exactly compute the proximal operator. For the nonconvex case (e.g. sorted Lq with q in ]0,1[), we show that a slight modification of this algorithm turns out to be remarkably efficient to tackle the computation of the proximal operator. We also present new theoretical insights on the minimizers of the nonconvex proximal problem. We demonstrate the practical interest of using such penalties on several experiments.

Problem

• Develop proximal operators for sorted nonconvex penalties
• Enable sparse signal recovery with automatic grouping
• Address weakly convex and nonconvex optimization challenges

Innovation

• Sorted nonconvex penalties for variable clustering
• PAV algorithm for weakly convex proximal operators
• Modified PAV for nonconvex proximal operator computation