Forgetting is Everywhere

2025-11-06

AI Summary

Catastrophic forgetting—the loss of previously acquired knowledge when adapting to new data—remains a fundamental challenge for general-purpose learning algorithms. Method: We propose the first algorithm- and task-agnostic unified forgetting theory, defining forgetting as the learner’s failure to maintain self-consistency in its predictive distribution over future experiences, quantified via predictive information loss. This framework unifies forgetting analysis across classification, regression, generative modeling, and reinforcement learning. We derive a computationally tractable information retention metric grounded in this theory. Contribution/Results: Extensive cross-paradigm experiments validate the metric’s ability to accurately characterize forgetting dynamics. Crucially, forgetting severity exhibits a strong negative correlation with learning efficiency. Our theory thus provides both a rigorous foundation for understanding forgetting and a practical tool for designing and evaluating forgetting-mitigation algorithms.

Technology Category

• Machine Learning: Information Theory
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

A fundamental challenge in developing general learning algorithms is their tendency to forget past knowledge when adapting to new data. Addressing this problem requires a principled understanding of forgetting; yet, despite decades of study, no unified definition has emerged that provides insights into the underlying dynamics of learning. We propose an algorithm- and task-agnostic theory that characterises forgetting as a lack of self-consistency in a learner's predictive distribution over future experiences, manifesting as a loss of predictive information. Our theory naturally yields a general measure of an algorithm's propensity to forget. To validate the theory, we design a comprehensive set of experiments that span classification, regression, generative modelling, and reinforcement learning. We empirically demonstrate how forgetting is present across all learning settings and plays a significant role in determining learning efficiency. Together, these results establish a principled understanding of forgetting and lay the foundation for analysing and improving the information retention capabilities of general learning algorithms.

Problem

• Developing general learning algorithms that avoid forgetting past knowledge
• Establishing unified theoretical framework to characterize forgetting dynamics
• Measuring forgetting across diverse learning settings and algorithms

Innovation

• Proposes algorithm-agnostic forgetting theory
• Defines forgetting as predictive inconsistency
• Introduces general measure for forgetting propensity