Efficient Long-Horizon Learning for Learned Optimization

2026-07-07

AI Summary

Existing learned optimizers struggle to achieve efficient meta-training on long-horizon inner-loop optimization problems and often underperform hand-designed counterparts. This work proposes ELO, an algorithm that substantially improves meta-training efficiency and stability by reallocating computational resources toward longer, previously failing optimization phases and incorporating a decoupled, progressive expert supervision mechanism. By integrating element-wise and matrix-wise optimizer architectures, ELO consistently outperforms well-tuned AdamW across language modeling and image classification tasks, approaching the performance of the state-of-the-art optimizer Muon. Notably, ELO achieves these results with less than 7 H100 GPU-hours of meta-training, demonstrating strong long-horizon optimization capabilities and robust out-of-distribution generalization.

Abstract

Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), current meta-training approaches still suffer from two main difficulties: (1) they cannot efficiently scale meta-training to long-horizon inner problems and (2) they often fail to surpass comparable hand-designed optimizers. To address these limitations, we propose Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates redundant meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. \textit{Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.}

Problem

• learned optimization
• long-horizon learning
• meta-training
• optimizer generalization
• inner-loop optimization

Innovation

• learned optimization
• efficient meta-training
• long-horizon learning
• progressive expert supervision
• out-of-distribution generalization