Non-Convex Sparse Reinforcement Learning via Non-Monotone Inclusions

2026-07-06

AI Summary

This work addresses the estimation bias induced by conventional regularization methods in reinforcement learning and the computational challenges posed by non-monotone inclusion problems. It introduces, for the first time, the weakly convex and sparsity-inducing proximal mapping of the capped-$\ell_1$ (PMC) regularizer into Least-Squares Temporal Difference (LSTD) learning for policy evaluation, formulating the problem as a non-monotone inclusion involving the sum of a monotone Lipschitz operator and a hypomonotone operator. To solve this formulation, the authors propose a forward-reflected-backward splitting algorithm and establish its convergence theory alongside a Lyapunov stability analysis. Experimental results on benchmark datasets with highly noisy features demonstrate that the proposed method significantly outperforms existing feature selection approaches, confirming its effectiveness and robustness.

Abstract

This work delivers two key contributions: one to efficient feature selection in reinforcement learning (RL), the other to the theory of non-monotone inclusions. On the RL side, the estimation bias inherent in conventional regularization schemes is addressed by augmenting classical least-squares temporal-difference (LSTD) policy evaluation with the sparsity-inducing, non-convex projected minimax concave (PMC) penalty. Because the PMC penalty is weakly convex, the resulting fixed-point problem is no longer monotone; instead, it falls under a broader class of non-monotone inclusions involving the sum of a monotone Lipschitz operator and a hypomonotone operator. On the theory side, novel convergence conditions are developed for the forward-reflected-backward splitting (FRBS) method applied to this broader class of non-monotone inclusion problems. Under mild conditions, Lyapunov stability and the existence of a limit point of the sequence of FRBS iterates are established; alternatively, under the weak Minty variational inequality assumption, exact convergence is guaranteed. Numerical tests on benchmark datasets show that the proposed FRBS iterates, applied to the non-convexly regularized LSTD problem, substantially outperform state-of-the-art feature-selection methods, especially when many noisy features are present.

Problem

• Sparse Reinforcement Learning
• Feature Selection
• Non-Convex Regularization
• Estimation Bias
• Non-Monotone Inclusions

Innovation

• non-convex regularization
• sparse reinforcement learning
• non-monotone inclusions
• forward-reflected-backward splitting
• projected minimax concave penalty