Reinforcement learning to choose optimizers

2026-09-01

AI Summary

研究通过强化学习选择优化器的方法，解决不同问题和阶段下最优优化方法的选择问题，采用序列决策制定策略以适应性选择最合适的优化器。

Abstract

No single optimization method is uniformly best for all problems, and the most suitable optimizer choice can change during a run. Existing approaches that change optimizer during execution typically predetermine part of the strategy: the portfolio is restricted to one algorithm class, the switch occurs once at a fixed time, or the frequency of decisions is treated as a hyperparameter rather than a learned one. We introduce "Reinforcement Learning to Choose Optimizers", which formulates the optimization algorithm choice as a sequential decision-making problem. At each decision, a recurrent policy reads the current run state and decides both which optimizer should be used next and for how long. The portfolio includes both gradient-based and derivative-free optimizers, and each switch passes on the current best solution and a representative step size. A context proxy conditions a gating network over expert heads, and training employs a decoupled actor-critic whose return is expressed in the same empirical runtime distribution metric used at evaluation. Training tasks and portfolio are designed jointly so that no optimizer dominates. On unseen problems, the learned policy outperforms every portfolio optimizer at all but the smallest budgets, and it remains robust under distribution shift.

Problem

• Reinforcement Learning
• Optimizers
• Sequential Decision-Making
• Gradient-based Optimizers
• Derivative-free Optimizers

Innovation

• Reinforcement Learning
• Sequential Decision Making
• Optimizer Portfolio
• Gating Network
• Decoupled Actor-Critic