FINDER: Stochastic Mirroring of Noisy Quasi-Newton Search and Deep Network Training

2024-10-18

AI Summary

Balancing noise-assisted global exploration and rapid local convergence remains challenging in high-dimensional nonconvex or nonsmooth optimization. To address this, we propose a derivative-free stochastic optimizer. Its core innovation lies in the first integration of a nonlinear stochastic filtering equation into the optimization update, enabling a quasi-Newton iteration that simultaneously captures inverse-Hessian geometry and maintains linear computational complexity. The method further incorporates adaptive noise injection, step-size adaptation, and physics-informed constraints. Extensive experiments on IEEE benchmark functions, deep learning tasks, and physics-informed neural networks (PINNs) demonstrate that our optimizer significantly outperforms mainstream methods—including Adam—achieving superior effectiveness and robustness on large-scale real-world problems.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Our proposal is on a new stochastic optimizer for non-convex and possibly non-smooth objective functions typically defined over large dimensional design spaces. Towards this, we have tried to bridge noise-assisted global search and faster local convergence, the latter being the characteristic feature of a Newton-like search. Our specific scheme -- acronymed FINDER (Filtering Informed Newton-like and Derivative-free Evolutionary Recursion), exploits the nonlinear stochastic filtering equations to arrive at a derivative-free update that has resemblance with the Newton search employing the inverse Hessian of the objective function. Following certain simplifications of the update to enable a linear scaling with dimension and a few other enhancements, we apply FINDER to a range of problems, starting with some IEEE benchmark objective functions to a couple of archetypal data-driven problems in deep networks to certain cases of physics-informed deep networks. The performance of the new method vis-'a-vis the well-known Adam and a few others bears evidence to its promise and potentialities for large dimensional optimization problems of practical interest.

Problem

• Develops a stochastic optimizer for non-convex, non-smooth functions.
• Bridges noise-assisted global search with faster local convergence.
• Applies FINDER to deep network training and physics-informed networks.

Innovation

• Stochastic optimizer for non-convex functions
• Derivative-free update resembling Newton search
• Linear scaling with dimension for efficiency