optimization

Generic convex optimizers suffer from poor scalability and inefficiency in adversarial training of linear models for large-scale problems. Method: This paper formulates linear adversarial robust learning as a convex optimization problem and introduces a dedicated solver based on extended-variable reparameterization: iterative ridge regression variants for regression tasks and projected gradient descent variants for classification tasks. Contribution/Results: The proposed approach significantly improves convergence speed and scalability, enabling efficient training on datasets with up to one million samples under rigorous theoretical guarantees. Numerical experiments demonstrate that the algorithms substantially outperform general-purpose convex solvers—including CVX and SCS—in both accuracy and computational speed, while maintaining theoretical soundness and practical deployability.

Neural network–embedded nonlinear optimization suffers from high-dimensional decision variables and slow convergence. Method: We propose a GPU-accelerated gray-box optimization framework that treats pre-trained neural networks as non-differentiable yet evaluable/differentiable “black-box constraints.” By modeling only the reduced input–output mapping—without exposing internal neurons or architecture—we drastically lower variable dimensionality; GPU-parallelized forward propagation and reverse-mode automatic differentiation are tightly integrated into an interior-point solver for efficient gradient computation. Contribution/Results: Evaluated on MNIST adversarial example generation and power system security-constrained dispatch, our method reduces iteration counts by 37% on average and achieves 2.1–3.8× speedup in total solve time, while preserving solution accuracy. It establishes a scalable paradigm for neuro-optimization co-modeling.

Mixed-integer linear programming (MILP)-based formal verification of neural networks suffers from computational intractability due to inherent nonlinearity and uncertainty in embedded models. Method: This paper proposes a sparse surrogate modeling approach based on neural network pruning: the original network is pruned into a low-precision, highly sparse subnetwork—termed a “surrogate-of-a-surrogate”—which is directly encoded into the MILP framework for adversarial verification. Contribution/Results: We establish, for the first time, that pruned networks with degraded classification accuracy are inherently more amenable to MILP optimization, significantly accelerating adversarial perturbation search. Crucially, no fine-tuning is required. Experiments demonstrate substantial speedups in verification runtime while preserving soundness and completeness. This work introduces an efficient, interpretable paradigm for constrained learning and formal verification of neural networks.

The rapid proliferation and evolution of deep learning optimizers necessitate a systematic synthesis to clarify their design principles, applicability domains, and shared challenges. Method: We propose the first unified mathematical framework for classifying and comparatively analyzing optimizers—from SGD and AdamW to recent methods such as Sophia and Muon—formally characterizing their update rules, hyperparameter semantics, and convergence properties. Contribution/Results: Through theoretical unification and empirical validation, we identify a coherent evolutionary trajectory: from adaptive learning rates, to second-order approximations, to sparse gradient correction. We explicitly delineate open challenges in generalization, large-model scalability, and computational efficiency. Our work establishes a structured knowledge base for optimizers, providing both theoretical foundations and practical guidance for algorithm selection, refinement, and novel paradigm design.

Learned optimizers (L2Os) suffer from poor out-of-distribution generalization, limiting their applicability beyond the training data distribution. Method: This paper proposes a novel paradigm integrating classical optimization priors with data-driven modeling. It systematically incorporates fundamental optimization principles—specifically scale invariance and affine covariance—into the architecture design. We introduce a parameterized quasi-Newton update module explicitly constrained to preserve BFGS structure, and jointly optimize it via end-to-end training that unifies optimization-theoretic modeling, neural network architecture design, and meta-learning. Contribution/Results: The resulting enhanced BFGS algorithm significantly outperforms both standard L2Os and conventional solvers on unseen problem classes, dimensions, and condition numbers. It achieves over 40% improvement in cross-distribution generalization performance, establishing a new pathway toward more transferable and robust learned optimizers.

Stochastic Gradient Descent (SGD) and its variants lack rigorous theoretical foundations in over-parameterized neural networks, suffering from inefficient training and poor interpretability. Method: This paper proposes a principle-driven guided descent framework that unifies, for the first time, curvature-aware second-order approximations, layer-adaptive preconditioning (calibrated via condition number), and a dynamically parameterized maximum-update learning rate mechanism. It systematically elucidates the synergistic interplay between this framework and exponential moving average (EMA) as well as learning rate scheduling. Contribution/Results: The method achieves both scalability and theoretical interpretability while preserving training stability and significantly accelerating convergence—reducing large-model training time by an order of magnitude. Moreover, it enhances discriminative feature learning, simultaneously improving generalization performance and output consistency.

This work addresses the multifaceted trade-offs faced by deep learning optimizers in convergence speed, generalization, computational efficiency, and privacy preservation. It presents the first unified framework that systematically elucidates the underlying principles and applicable scenarios of first-order (e.g., SGD, Adam), second-order, and zeroth-order optimization methods. Through large-scale empirical evaluations across diverse model architectures and practical constraints—including distributed training and differential privacy—the study combines theoretical analysis with extensive experimentation to distill fundamental trade-offs and evolutionary patterns in optimizer design. Building on these insights, the paper proposes actionable guidelines for developing optimizers that enable efficient, robust, and trustworthy training. The accompanying benchmarking codebase is publicly released to foster reproducibility and further research.

This work addresses the limited convergence rate of the BFGS algorithm in quasi-Newton methods by proposing a novel search direction termed the quasi-quadratic gradient (QQG). The QQG method explicitly incorporates local second-order curvature information into the construction of the search direction for the first time, achieved by dynamically correcting the optimization trajectory through multiplication of the inverse Hessian approximation with the current gradient. While preserving the computational efficiency inherent to BFGS, this approach significantly accelerates convergence. Numerical experiments demonstrate that QQG achieves faster convergence rates than standard BFGS across a variety of benchmark problems, thereby validating the effectiveness and superiority of explicitly leveraging local curvature information in the design of search directions.

Addressing the challenges of Hessian estimation—namely, high computational cost and numerical instability—in high-dimensional nonconvex optimization, this paper proposes Online Gradient Regression (OGR). OGR models the local second-order relationship between gradients and parameter displacements via exponential moving averages, enabling direct estimation of general (not necessarily positive-definite) Hessian matrices—thereby relaxing the positive-definiteness requirement inherent in BFGS-type methods. By integrating Sherman–Morrison updates with an online learning framework, OGR avoids explicit Hessian storage and inversion, significantly improving both computational efficiency and numerical robustness. Experiments on standard nonconvex benchmark functions demonstrate that OGR achieves faster convergence and lower final loss than BFGS, especially in high-dimensional settings. This work establishes a new, scalable, and robust paradigm for Hessian approximation in second-order optimization of neural networks.

To address the challenge of solving parametric nonlinear constrained optimization problems at high frequency in real-time control and model-based design, this paper proposes an end-to-end neural network that directly learns the mapping from problem parameters to both primal and dual variables. Our method innovatively incorporates the Karush–Kuhn–Tucker (KKT) optimality residual into the loss function and employs constraint-aware output activations (e.g., Softplus or Clamp) to intrinsically enforce feasibility and optimality during training—thereby reducing data dependency and enabling high-accuracy dual variable prediction. Experiments demonstrate that, compared to a quadratic penalty baseline, our approach reduces constraint violations by 37%, decreases primal variable error by 22%, achieves dual variable prediction error below 0.05, and exhibits superior hyperparameter robustness.