ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think

2025-01-02

AI Summary

This work addresses the challenge of evaluating and mitigating catastrophic forgetting in continual learning when gradient information is unavailable. To this end, we propose ZeroFlow—the first gradient-free benchmark for continual learning. Methodologically, we establish, for the first time, that forward propagation alone inherently possesses forgetting-suppression capability; leveraging this insight, we design a single-forward-pass activation mechanism and introduce a novel forward-only optimization paradigm that jointly manages task interference and ensures memory efficiency. Extensive experiments across multiple datasets and forgetting scenarios demonstrate that our purely forward-based approach matches or surpasses backward-propagation-based methods in accuracy, while significantly reducing memory footprint and hardware dependency. ZeroFlow is open-sourced, providing a standardized evaluation platform and theoretical foundation for gradient-free continual learning.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Backpropagation provides a generalized configuration for overcoming catastrophic forgetting. Like, SGD and Adam are commonly used for weight updates in continual learning and continual pre-training. In practice, permission to access gradient information is not always granted (the gradient ban), such as black-box APIs, hardware limitations, and non-differentiable systems. To bridge this gap, we introduce the first benchmark ZeroFlow to evaluate gradient-free optimization algorithms for overcoming forgetting. This benchmark examines a suite of forward pass methods across multiple methods, forgetting scenarios, and datasets. We find that forward passes alone are enough to overcome forgetting. Our findings reveal new optimization principles that highlight the potential of forward-pass in mitigating forgetting, managing task conflicts, and reducing memory demands, alongside novel enhancements that further mitigate forgetting with just one forward pass. This work provides essential insights and tools for advancing forward pass methods to overcome forgetting.

Problem

• Continual Learning
• Gradient Information
• Knowledge Retention

Innovation

• ZeroFlow
• Forward Propagation
• Superior Forgetfulness Mitigation