The Gentle Collapse: Distributional Metrics for Continual Learning

2026-06-23

AI Summary

This work addresses the limitation of traditional accuracy metrics in continual learning, which only provide a binary indication of catastrophic forgetting and fail to capture its internal structure. The authors propose six continuous distribution-based measures derived from softmax outputs—such as ground-truth label rank, prediction confidence, and distributional divergence—that enable fine-grained quantification of forgetting as a value within the [0,1] interval. These metrics reveal semantic differences even when accuracy drops to zero, without requiring modifications to the training procedure. Leveraging these measures, the study introduces a weighted experience replay mechanism and a trend-slope-based sample selection strategy. Experiments demonstrate that the proposed approach reduces forgetting by 1.3% on CIFAR-100 and by 7.7% on TinyImageNet compared to baseline methods, significantly outperforming existing techniques that rely solely on accuracy trends.

Abstract

Accuracy degradation is the standard metric for Catastrophic Forgetting (CF), however, it records only whether forgetting occurred or not. It saturates at the extremes and collapses discretely at task boundaries, hiding the internal structure of what is being forgotten. We introduce six softmax-derived metrics spanning true-label rank (TLR), predictive confidence, and distributional divergence that characterize forgetting continuously, each normalized to [0, 1] with no modification to training. On CIFAR-100, these metrics carry information where accuracy does not: at 0% accuracy, the Confusion Margin spans an IQR of [0.32, 0.50] across classes that accuracy treats identically. We demonstrate that this richer signal is actionable in mitigating catastrophic forgetting. Per-sample metric scores used as loss weights reduce forgetting by 1.3 percentage points over uniform experience replay (ER) on CIFAR-100. Furthermore, the slope of a metric over a small window provides a stable sampling criterion: at a small-window size (e.g. 3 epochs), accuracy-trend degrades to 34.79% (std. = 2.32) while log-TLR achieves 41.07% (std. = 0.57). This gap is structural since reliable small-window trend estimation requires a continuous signal. On TinyImageNet, log-TLR trend sampling reduces forgetting by 7.7 percentage points over the ER baseline.

Problem

• Catastrophic Forgetting
• Continual Learning
• Distributional Metrics
• Accuracy Degradation
• Forgetting Characterization

Innovation

• continual learning
• catastrophic forgetting
• distributional metrics
• softmax-derived metrics
• experience replay