Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

2026-05-19

AI Summary

This work addresses catastrophic forgetting in large language model fine-tuning, which often degrades pre-trained knowledge. The authors propose FINCH, a loss-adaptive learning rate scheduling method that requires no modification to the optimization objective. By uncovering a theoretical link between one-step forgetting and the product of the learning rate and the square root of the loss, FINCH dynamically adjusts the learning rate—reducing it under high loss and gradually increasing it as training converges—to effectively mitigate forgetting. Experiments demonstrate that FINCH reduces knowledge forgetting by 93% on average across multiple benchmarks while matching the task performance of standard fine-tuning. Notably, on Qwen3-4B, it reduces TruthfulQA degradation by fivefold, reverses performance decline on HaluEval, and better preserves confidence calibration.

Abstract

Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining, a phenomenon known as catastrophic forgetting. Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences, but these tokens are essential for learning new tasks, especially those with poor pretraining coverage. In such settings, hard tokens should still contribute to learning, so forgetting must be controlled without suppressing them. We identify a simple mechanism for doing so: per-step forgetting is bounded by the product of the learning rate and the square root of the current training loss. This suggests that high-loss batches are especially prone to inducing forgetting. Motivated by this observation, we introduce FINCH, a loss-adaptive learning-rate schedule that reduces the learning rate on high-loss batches and increases it as the model converges, while leaving the fine-tuning objective unchanged. Across knowledge acquisition, science, and low-resource language adaptation benchmarks, FINCH reduces forgetting by 93% on average while matching the task performance of standard fine-tuning. On Qwen3-4B knowledge acquisition, FINCH cuts TruthfulQA degradation by 5x and reverses HaluEval degradation, while better preserving confidence calibration. Overall, our results show that learning-rate schedules are an effective tool to shape model behavior during fine-tuning, beyond just target-task optimization.

Problem

• catastrophic forgetting
• fine-tuning
• large language models
• loss-adaptive learning rates
• knowledge retention

Innovation

• catastrophic forgetting
• loss-adaptive learning rate
• fine-tuning
• large language models
• FINCH