The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models

2025-12-26

AI Summary

Parameter-efficient fine-tuning (PEFT) methods such as LoRA often induce catastrophic forgetting of pre-trained general capabilities in large language models during instruction tuning—especially under few-shot settings. To address this, we propose Approximate Regularized Replay (ARR), the first approach that jointly leverages KL-divergence regularization and lightweight, cross-corpus next-token replay. ARR interleaves samples from multiple data sources to simultaneously optimize knowledge retention and task adaptation with minimal overhead. Evaluated on the Qwen family of models, ARR preserves new-task performance while substantially mitigating capability degradation; pre-trained knowledge integrity improves markedly, with only ~15% additional computational cost. This work provides an efficient, general-purpose, and deployment-friendly solution to the continual learning challenge in PEFT.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Although parameter-efficient fine-tuning methods, such as LoRA, only modify a small subset of parameters, they can have a significant impact on the model. Our instruction-tuning experiments show that LoRA-based supervised fine-tuning can catastrophically degrade model capabilities, even when trained on very small datasets for relatively few steps. With that said, we demonstrate that while the most straightforward approach (that is likely the most used in practice) fails spectacularly, small tweaks to the training procedure with very little overhead can virtually eliminate the problem. Particularly, in this paper we consider a regularized approximate replay approach which penalizes KL divergence with respect to the initial model and interleaves in data for next token prediction from a different, yet similar, open access corpus to what was used in pre-training. When applied to Qwen instruction-tuned models, we find that this recipe preserves general knowledge in the model without hindering plasticity to new tasks by adding a modest amount of computational overhead.

Problem

• Addresses catastrophic degradation in model capabilities during LoRA-based fine-tuning.
• Proposes regularized replay to preserve knowledge without hindering task plasticity.
• Mitigates performance loss with minimal computational overhead in supervised fine-tuning.

Innovation

• Regularized replay penalizes KL divergence from initial model
• Interleaves next token prediction data from similar corpus
• Preserves general knowledge while enabling new task plasticity