Constrained Variable Projection for Structured Problems

2026-06-22

AI Summary

This work addresses structured separable nonlinear least squares problems where a subset of variables is subject to convex constraints. Viewing variable projection as a special case of bilevel optimization, the paper extends this approach for the first time to constrained settings by introducing a constrained variable projection framework. The authors derive an exact reduced gradient formula compatible with automatic differentiation and develop an efficient conditional gradient-based algorithm with convergence guarantees. Empirical evaluations on tasks including sparse autoencoding, dictionary learning, blind deconvolution, and few-shot learning demonstrate that the proposed method substantially improves computational efficiency and data utilization compared to joint optimization baselines.

Abstract

Variable projection is a classical technique for separable nonlinear least-squares problems, in which variables that enter linearly are eliminated exactly, yielding a reduced nonlinear problem. By expressing this framework as a particular instance of a broader class of bilevel optimization problems, we develop a constrained variable-projection framework for data-science models, where the remaining variables are subject to convex constraints and the eliminated variables arise from a lower-level least-squares problem. In particular, by interpreting variable projection as a collapsed bilevel optimization problem, we derive exact reduced-gradient formulas compatible with automatic differentiation and propose a conditional-gradient algorithm for the resulting constrained reduced problem. We establish convergence guarantees under standard smoothness and compactness assumptions, and discuss extensions to structured lower-level variables. Numerical experiments on sparse autoencoding, dictionary learning, blind deconvolution, and few-shot learning suggest that the method can improve wall-clock efficiency and data efficiency relative to natural joint-optimization baselines.

Problem

• variable projection
• bilevel optimization
• convex constraints
• structured problems
• nonlinear least squares

Innovation

• constrained variable projection
• bilevel optimization
• reduced gradient
• conditional gradient algorithm
• structured least-squares