Local LMO: Constrained Gradient Optimization via a Local Linear Minimization Oracle

2026-05-09

AI Summary

This work addresses the limitations of the classical Frank-Wolfe method, which relies on a global linear minimization oracle (LMO) and requires bounded feasible sets along with curvature assumptions. The authors propose a Local LMO algorithm that solves a local linear minimization problem over the intersection of a neighborhood around the current iterate and the constraint set, thereby enabling projection-free gradient-type optimization. This approach extends the convergence theory of projected gradient descent to projection-free settings, eliminating the need for boundedness and curvature conditions. It provides a unified framework for convex, strongly convex, nonconvex, and stochastic optimization problems. Under various settings, the algorithm achieves optimal convergence rates matching those of projected gradient descent: sublinear for convex objectives, linear for smooth strongly convex cases, and optimal sublinear rates for nonconvex, stochastic, and nonsmooth scenarios.

Abstract

We design Local LMO - a new projection-free gradient-type method for constrained optimization. The key algorithmic idea is to replace the global linear minimization oracle over the constraint set used by Frank-Wolfe (FW) with a local linear minimization oracle over the intersection of the constraint set and a "small" ball centered at the current iterate. In particular, when minimizing $f:\mathbb{R}^d\to \mathbb{R}$ over a constraint $\emptyset\neq\mathcal{X}\subseteq\mathbb{R}^d$, Local LMO performs the iteration \[x_{k+1}\in \arg\min_{z\in\mathcal{X}\cap\mathcal{B}(x_{k},t_k)}\langle\nabla f(x_{k}), z \rangle,\] where $x_0\in\mathcal{X}$, and $t_k>0$ is a suitably chosen radius which can be interpreted as an effective stepsize. While designed as an alternative to FW, Local LMO is perhaps best viewed as a generalization of Gradient Descent (GD) rather than a modification of FW. Indeed, it is easy to see that Local LMO reduces to GD in the unconstrained setting and, more generally, to GD restricted to an affine subspace if the constraint $\mathcal{X}$ is affine. We prove that this simple algorithmic scheme transfers the known (unaccelerated) convergence rates of Projected Gradient Descent (PGD) to the projection-free world in several important regimes, some of which are beyond the reach of FW. In contrast to FW theory, i) our guarantees hold without requiring the feasible set $\mathcal{X}$ to be bounded, ii) our theory does not require the "curvature" assumption, which allows us to establish a standard sublinear rate for convex functions with bounded gradients, iii) we obtain a linear rate in the smooth strongly convex regime. Furthermore, we obtain sharp sublinear rates in the smooth convex and non-convex regimes, in the $(L_0,L_1)$-smooth convex regime, and in stochastic and non-differentiable settings.

Problem

• constrained optimization
• projection-free
• gradient methods
• convergence rates
• linear minimization oracle

Innovation

• Local LMO
• projection-free optimization
• constrained optimization
• gradient descent generalization
• linear minimization oracle