Online Curvature-Aware Replay: Leveraging $mathbf{2^{nd}}$ Order Information for Online Continual Learning

2025-02-03

AI Summary

In online continual learning (OCL) on non-stationary data streams with unknown task boundaries, existing methods suffer from severe forgetting and unstable optimization due to ineffective replay. To address this, we propose a second-order online joint optimization framework constrained by KL divergence. Our approach is the first to incorporate the K-FAC-approximated Fisher information matrix into OCL, enabling curvature-aware gradient preconditioning. We further introduce a dynamic Tikhonov regularization mechanism that explicitly balances stability and plasticity. The method integrates Fisher-guided gradient correction, KL-constrained optimization, and adaptive second-order parameter estimation. Evaluated on three standard benchmarks, it achieves statistically significant improvements in average accuracy over state-of-the-art methods, exhibits enhanced training stability, substantially mitigates catastrophic forgetting, and accelerates convergence.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Online Continual Learning (OCL) models continuously adapt to nonstationary data streams, usually without task information. These settings are complex and many traditional CL methods fail, while online methods (mainly replay-based) suffer from instabilities after the task shift. To address this issue, we formalize replay-based OCL as a second-order online joint optimization with explicit KL-divergence constraints on replay data. We propose Online Curvature-Aware Replay (OCAR) to solve the problem: a method that leverages second-order information of the loss using a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient. The FIM acts as a stabilizer to prevent forgetting while also accelerating the optimization in non-interfering directions. We show how to adapt the estimation of the FIM to a continual setting stabilizing second-order optimization for non-iid data, uncovering the role of the Tikhonov regularization in the stability-plasticity tradeoff. Empirical results show that OCAR outperforms state-of-the-art methods in continual metrics achieving higher average accuracy throughout the training process in three different benchmarks.

Problem

• Online Continual Learning adaptation
• Second-order optimization stabilization
• Preventing forgetting in nonstationary data

Innovation

• Second-order online joint optimization
• K-FAC approximation of FIM
• Tikhonov regularization stability-plasticity tradeoff