Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm

2026-05-11

AI Summary

This work addresses the challenge of catastrophic forgetting in language models during continual pretraining, which hinders effective acquisition and retention of factual knowledge. The authors develop a theoretical framework based on a single-layer Transformer to unify the understanding of representative mitigation strategies: they prove that regularization merely modulates convergence speed without preventing forgetting, whereas data replay alters convergence dynamics to stabilize previously learned knowledge. Building on this insight, they propose STOC, a novel paradigm that identifies critical factual segments by analyzing attention contributions and leverages them to guide generative replay. Experiments on both synthetic and real-world datasets validate the theoretical analysis and demonstrate that STOC substantially mitigates forgetting and enhances the model’s ability to continually acquire factual knowledge.

Abstract

Continual Pre-Training (CPT) is essential for enabling Language Models (LMs) to integrate new knowledge without erasing old. While classical CPT techniques like data replay have become the standard paradigm, the mechanisms underlying how LMs acquire and retain facts over time, termed as continual Factual Knowledge Acquisition (cFKA), remain unclear. In this work, we present a theoretical framework that characterizes the training dynamics of cFKA using a single-layer Transformer, offering a unified explanation for the behavior of representative CPT methods. Our analysis reveals that regularization-based methods merely adjust the convergence rate of parameters without altering the inherent forgetting tendency, whereas data replay methods succeed in shifting convergence dynamics and stabilizing pretrained knowledge. Building on these insights, we propose a novel generative data replay approach, called \textbf{S}electing \textbf{T}okens via attenti\textbf{O}n \textbf{C}ontribution~(STOC), which identifies influential factual snippets to guide replay data generation. Extensive experiments on both synthetic and real-world datasets validate our findings and demonstrate that STOC effectively enhances cFKA by mitigating catastrophic forgetting.

Problem

• continual learning
• factual knowledge acquisition
• catastrophic forgetting
• language models
• continual pre-training

Innovation

• continual factual knowledge acquisition
• theoretical framework
• data replay
• attention contribution
• catastrophic forgetting