ErrorEraser: Unlearning Data Bias for Improved Continual Learning

2025-06-11

AI Summary

In continual learning (CL), real-world data biases induce spurious correlations that propagate and amplify across task sequences, exacerbating catastrophic forgetting and impairing knowledge transfer. To address this, we propose *conscious forgetting*, a novel paradigm that automatically identifies and actively discards bias-laden samples without requiring prior domain knowledge. Our approach leverages probabilistic density estimation and anomaly detection to localize biased instances, corrects decision boundaries to erase associated errors, and employs incremental feature distribution modeling to disentangle bias from semantic representations. The method is framework-agnostic—seamlessly integrating into replay-, regularization-, and parameter-isolation-based CL strategies. Evaluated on multiple benchmarks, it consistently reduces forgetting rates by an average of 12.3%, while simultaneously improving both backward transfer (old-task retention) and forward transfer (new-task accuracy). This work is the first to systematically incorporate bias-aware forgetting as a core mechanism in continual learning.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Bias, Fairness & Privacy
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Continual Learning (CL) primarily aims to retain knowledge to prevent catastrophic forgetting and transfer knowledge to facilitate learning new tasks. Unlike traditional methods, we propose a novel perspective: CL not only needs to prevent forgetting, but also requires intentional forgetting.This arises from existing CL methods ignoring biases in real-world data, leading the model to learn spurious correlations that transfer and amplify across tasks. From feature extraction and prediction results, we find that data biases simultaneously reduce CL's ability to retain and transfer knowledge. To address this, we propose ErrorEraser, a universal plugin that removes erroneous memories caused by biases in CL, enhancing performance in both new and old tasks. ErrorEraser consists of two modules: Error Identification and Error Erasure. The former learns the probability density distribution of task data in the feature space without prior knowledge, enabling accurate identification of potentially biased samples. The latter ensures only erroneous knowledge is erased by shifting the decision space of representative outlier samples. Additionally, an incremental feature distribution learning strategy is designed to reduce the resource overhead during error identification in downstream tasks. Extensive experimental results show that ErrorEraser significantly mitigates the negative impact of data biases, achieving higher accuracy and lower forgetting rates across three types of CL methods. The code is available at https://github.com/diadai/ErrorEraser.

Problem

• Addresses data bias in continual learning systems
• Reduces spurious correlations across sequential tasks
• Enhances knowledge retention and transfer capabilities

Innovation

• ErrorEraser removes bias-induced erroneous memories
• Identifies biased samples via density distribution
• Erases errors by shifting decision space