Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution

2026-04-29

AI Summary

This work addresses the challenge in transfer learning of designing learning rate schedules that simultaneously preserve low-level generalizable knowledge and enable high-level task adaptation. To this end, the authors propose DALS, a deep adaptive learning rate scheduling framework that jointly optimizes learning rates across network layers and training phases by integrating phase-adaptive cosine annealing, depth-aware Grokfast gradient filtering, and the LARS trust ratio mechanism. As the first unified approach to systematically combine these techniques, DALS achieves 98.0% accuracy on synthetic benchmarks, with its accelerated variant DALS-Fast reaching 90% accuracy in just three training epochs. The method demonstrates state-of-the-art or competitive performance across diverse scenarios, including cross-dataset training from scratch and fine-tuning tasks.

Abstract

Learning rate scheduling has evolved from the single global fixed rate of early SGD to sophisticated layer-wise adaptive strategies. We systematize this evolution into five generations: (Gen1) global fixed learning rates, (Gen2) global scheduling, (Gen3) parameter-level adaptation, (Gen4) layer-level differentiation, and (Gen5) joint layer-time scheduling. We trace the fundamental motivation behind each transition, showing how the shift from one-size-fits-all to tailoring by layer and time addresses the impossible trinity of transfer learning: lower layers require small updates to preserve general knowledge while higher layers need large updates to adapt to new tasks. Building on this taxonomy, we propose Discriminative Adaptive Layer Scaling (DALS), a unified framework that integrates phase-adaptive cosine scheduling, depth-aware Grokfast gradient filtering, and LARS-style trust ratios into a single coherent optimizer. We benchmark 18 strategies including three DALS variants across all five generations on five datasets: synthetic, CIFAR-10 (from scratch), RTE, TREC-6, and IMDb (fine-tuning). On synthetic, DALS achieves the best accuracy at 98.0%, while DALS-Fast reaches 90% in just 3 epochs. The cross-dataset analysis reveals striking regime-dependent patterns -- no single strategy wins across all regimes. Critically, STLR+Discriminative, the ULMFiT champion, catastrophically fails on from-scratch tasks (43.6% on TREC-6 from scratch vs. 96.8% with RAdam), confirming that directional decay biases are harmful without pretrained features. DALS avoids either extreme, achieving the best synthetic result while maintaining competitive fine-tuning performance.

Problem

• learning rate scheduling
• layer-wise adaptation
• transfer learning
• optimization
• deep neural networks

Innovation

• Discriminative Adaptive Layer Scaling
• layer-wise learning rate
• learning rate scheduling
• depth-aware optimization
• joint layer-time scheduling