First-Order Methods for Linearly Constrained Bilevel Optimization

2024-06-18

Neural Information Processing Systems

AI Summary

This work addresses the infeasibility of Hessian computation in high-dimensional linearly constrained (both equality and inequality) bilevel optimization. Methodologically, it introduces the first first-order algorithmic framework for such problems, establishing the inaugural first-order theoretical foundation for linearly constrained bilevel optimization. A novel Goldstein stationarity paradigm is proposed to handle nonsmooth inequality constraints, while dual-variable embedding and projection techniques ensure dimension-independent complexity. Theoretically, under equality constraints, the hypergradient converges at rate Õ(ε⁻²) in gradient evaluations; under inequality constraints, oracle complexities of Õ(dδ⁻¹ε⁻³) and dimension-independent Õ(δ⁻¹ε⁻⁴) are achieved, respectively. Numerical experiments validate both the algorithm’s efficacy and the tightness of the theoretical bounds.

Technology Category

• Search and Optimization: Non-convex Optimization
• Constraint Satisfaction and Optimization: Constraint Optimization
• Machine Learning: Optimization

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Algorithms for bilevel optimization often encounter Hessian computations, which are prohibitive in high dimensions. While recent works offer first-order methods for unconstrained bilevel problems, the constrained setting remains relatively underexplored. We present first-order linearly constrained optimization methods with finite-time hypergradient stationarity guarantees. For linear equality constraints, we attain $epsilon$-stationarity in $widetilde{O}(epsilon^{-2})$ gradient oracle calls, which is nearly-optimal. For linear inequality constraints, we attain $(delta,epsilon)$-Goldstein stationarity in $widetilde{O}(d{delta^{-1} epsilon^{-3}})$ gradient oracle calls, where $d$ is the upper-level dimension. Finally, we obtain for the linear inequality setting dimension-free rates of $widetilde{O}({delta^{-1} epsilon^{-4}})$ oracle complexity under the additional assumption of oracle access to the optimal dual variable. Along the way, we develop new nonsmooth nonconvex optimization methods with inexact oracles. We verify these guarantees with preliminary numerical experiments.

Problem

• Develop first-order methods for linearly constrained bilevel optimization
• Achieve near-optimal stationarity guarantees for equality constraints
• Provide dimension-free rates for inequality constraints with dual access

Innovation

• First-order methods for constrained bilevel optimization
• Finite-time hypergradient stationarity guarantees
• Dimension-free rates with optimal dual variable