Convergence of Consensus-Based Particle Methods for Nonconvex Bi-Level Optimization

2026-05-19

AI Summary

This work addresses nonconvex bilevel optimization problems, where the upper-level objective is minimized over the set of global minimizers of a lower-level problem. The authors propose a derivative-free particle consensus optimization method that constructs consensus points by combining smoothed quantile selection with a Gibbs-type Laplace approximation. For the first time, they establish a particle-based convergence theory for nonconvex bilevel optimization. Through mean-field limit analysis, they prove that the algorithm converges to the target solution at an exponential rate in hitting time and provide an explicit error bound within a Wasserstein neighborhood. Numerical experiments demonstrate the method's effectiveness on both two-dimensional constrained problems and neural network training tasks.

Abstract

In this paper, we study a consensus-based optimization method for nonconvex bi-level optimization, where the objective is to minimize an upper-level function over the set of global minimizers of a lower-level problem. The proposed approach is derivative-free, and constructs its consensus point via smooth quantile selection combined with a Gibbs-type Laplace approximation. We establish convergence guarantees for both the associated \textit{mean-field} dynamics and its \textit{finite-particle} approximation. In particular, under suitable assumptions on smooth quantile localization, error bounds, and stability, we show that the mean-field law reaches any arbitrary prescribed Wasserstein neighborhood of the target bi-level solution with an explicit exponential rate up to the hitting time. Numerical experiments on a two-dimensional constrained problem and neural network training further support the theoretical results.

Problem

• nonconvex bi-level optimization
• consensus-based optimization
• derivative-free
• mean-field dynamics
• Wasserstein convergence

Innovation

• consensus-based optimization
• bilevel optimization
• mean-field dynamics
• smooth quantile selection
• derivative-free