Can Scale Save Us From Plasticity Loss in Large Language Models?

2026-06-23

AI Summary

This study addresses the susceptibility of large language models to loss of plasticity during continual learning—specifically, their diminished capacity to effectively integrate new knowledge while retaining previously acquired information. By systematically training GPT-style Transformer models with non-embedding parameter counts ranging from 5M to 314M under both multilingual continual learning and static training regimes, the authors evaluate model plasticity using a Vietnamese probing task. They provide the first empirical evidence of plasticity loss in modern large language models, demonstrating that this phenomenon occurs not only under abrupt task shifts but also during static multilingual training. Although increasing model scale mitigates plasticity loss sublinearly, it does not prevent it entirely, indicating that even large models eventually lose the ability to adapt efficiently to new data after prolonged training.

Abstract

The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer language models trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.

Problem

• plasticity loss
• continual learning
• large language models
• Transformer
• multilingual training

Innovation

• plasticity loss
• continual learning
• scaling law
• Transformer language models
• multilingual training