Statistical Inference of Constrained Stochastic Optimization via Sketched Sequential Quadratic Programming

2022-05-27

AI Summary

This paper addresses constrained stochastic nonlinear optimization problems arising in online statistical inference. We propose Sketch-StoSQP, a sketched stochastic sequential quadratic programming method. Our key contributions are threefold: (i) We establish, for the first time, the asymptotic normality of StoSQP iterates under controllable, non-vanishing approximation errors—ensuring stable per-iteration computational complexity; (ii) We design a plug-and-play covariance estimator enabling immediate statistical inference without algorithmic modification; (iii) We prove that the scaled residual sequence converges in distribution to a non-degenerate zero-mean Gaussian. Empirical evaluation on the CUTEst benchmark and constrained regression tasks demonstrates both statistical validity—accurate coverage rates and well-calibrated confidence intervals—and computational efficiency—constant per-iteration cost and significant overall speedup.

Abstract

We consider online statistical inference of constrained stochastic nonlinear optimization problems. We apply the Stochastic Sequential Quadratic Programming (StoSQP) method to solve these problems, which can be regarded as applying second-order Newton's method to the Karush-Kuhn-Tucker (KKT) conditions. In each iteration, the StoSQP method computes the Newton direction by solving a quadratic program, and then selects a proper adaptive stepsize $ar{alpha}_t$ to update the primal-dual iterate. To reduce dominant computational cost of the method, we inexactly solve the quadratic program in each iteration by employing an iterative sketching solver. Notably, the approximation error of the sketching solver need not vanish as iterations proceed, meaning that the per-iteration computational cost does not blow up. For the above StoSQP method, we show that under mild assumptions, the rescaled primal-dual sequence $1/sqrt{ar{alpha}_t}cdot (x_t - x^star, lambda_t - lambda^star)$ converges to a mean-zero Gaussian distribution with a nontrivial covariance matrix depending on the underlying sketching distribution. To perform inference in practice, we also analyze a plug-in covariance matrix estimator. We illustrate the asymptotic normality result of the method both on benchmark nonlinear problems in CUTEst test set and on linearly/nonlinearly constrained regression problems.

Problem

• Online statistical inference
• Constrained stochastic optimization
• Sketching Sequential Quadratic Programming

Innovation

• Stochastic Sequential Quadratic Programming
• Iterative sketching solver
• Plug-in covariance matrix estimator