Optimal Restart Strategies for Parameter-dependent Optimization Algorithms
This work addresses the challenge of adaptively selecting the unknown optimal regularization parameter λ in parameter-dependent optimization algorithms, where excessively large λ incurs prohibitive computational cost while overly small λ yields low success probability. We propose the first classification framework for restart strategies based on bounded relative loss. Theoretically, we prove that multiplicative growth schemes admit an asymptotically optimal scaling factor independent of the true λ. Through rigorous parameter sensitivity analysis, worst-case modeling, and derivation of tight upper and lower bounds on relative loss, we establish boundedness of the relative loss under this strategy and derive an explicit closed-form optimal scaling factor that minimizes the worst-case relative loss. Crucially, this factor’s asymptotic optimality is agnostic to the unknown λ, thereby significantly enhancing both restart efficiency and robustness.