Accelerating Optimization via Differentiable Stopping Time

2025-05-28

AI Summary

This work addresses the challenge of hyperparameter optimization arising from the non-differentiability of “time-to-target-loss”—a key metric in iterative optimization. We propose the first differentiable stopping-time framework. Methodologically, we model optimization trajectories via ordinary differential equations (ODEs), then leverage the implicit function theorem and extended backpropagation to obtain an accurate, differentiable estimate of the first hitting time to a target loss; adaptive numerical integration ensures stability. Theoretically, we provide the first rigorous proof of differentiability for this stopping time, overcoming the long-standing limitation posed by nonsmooth termination criteria. Empirically, our framework accelerates convergence significantly: it achieves up to 3.2× higher online hyperparameter tuning efficiency across diverse tasks and surpasses zero-order methods and existing differentiable optimization baselines in generalization performance.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.

Problem

• Differentiable stopping time for optimization acceleration
• Minimizing time to reach target loss differentiably
• Enabling online hyperparameter tuning via differentiable framework

Innovation

• Differentiable stopping time for optimization
• Backpropagation through stopping time
• Online hyperparameter tuning application