The Online Patch Redundancy Eliminator (OPRE): A novel approach to online agnostic continual learning using dataset compression

2025-11-11

AI Summary

Addressing the dual challenges of catastrophic forgetting and reliance on prior knowledge in continual learning, this paper proposes OPRE, a prior-free online continual learning framework. Its core is an online patch redundancy elimination compression algorithm, which operates under minimal data assumptions—such as local smoothness—and for the first time systematically reveals the fundamental limitations of pretrained feature extractors on continual learning generalizability. OPRE dispenses with task identifiers, replay buffers, and domain priors; instead, it dynamically compresses redundant information upon data stream arrival and enables plug-and-play learning via test-time classifier adaptation. Evaluated on standard CIFAR-10/100 benchmarks, OPRE consistently outperforms mainstream replay-based, regularization-based, and parameter-isolation methods. These results validate the effectiveness and scalability of the “prior-free + online compression” paradigm for continual learning.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In order to achieve Continual Learning (CL), the problem of catastrophic forgetting, one that has plagued neural networks since their inception, must be overcome. The evaluation of continual learning methods relies on splitting a known homogeneous dataset and learning the associated tasks one after the other. We argue that most CL methods introduce a priori information about the data to come and cannot be considered agnostic. We exemplify this point with the case of methods relying on pretrained feature extractors, which are still used in CL. After showing that pretrained feature extractors imply a loss of generality with respect to the data that can be learned by the model, we then discuss other kinds of a priori information introduced in other CL methods. We then present the Online Patch Redundancy Eliminator (OPRE), an online dataset compression algorithm, which, along with the training of a classifier at test time, yields performance on CIFAR-10 and CIFAR-100 superior to a number of other state-of-the-art online continual learning methods. Additionally, OPRE requires only minimal and interpretable hypothesis on the data to come. We suggest that online dataset compression could well be necessary to achieve fully agnostic CL.

Problem

• Overcoming catastrophic forgetting in neural networks for continual learning
• Developing agnostic methods without prior data assumptions for CL
• Using online dataset compression to improve performance in CL

Innovation

• Online dataset compression algorithm for continual learning
• Training classifier at test time for improved performance
• Minimal interpretable hypotheses for agnostic continual learning