Escaping Stability-Plasticity Dilemma in Online Continual Learning for Motion Forecasting via Synergetic Memory Rehearsal

2025-08-27

AI Summary

To address the stability–plasticity dilemma in online continual learning for motion prediction—where models readily forget previously acquired knowledge when adapting to new driving scenarios—this paper proposes SyReM. The method introduces three key components: (1) inequality constraints on memory replay loss increments to preserve historical knowledge stability; (2) an online gradient analysis mechanism based on cosine similarity to enable selective replay of representative historical samples, thereby enhancing plasticity; and (3) a compact memory buffer for efficient online model updates. Extensive experiments across 11 real-world driving datasets demonstrate that SyReM significantly mitigates catastrophic forgetting. It achieves superior motion prediction accuracy on both newly encountered and previously learned driving scenarios, outperforming both non-continual learning baselines and state-of-the-art continual learning approaches.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Deep neural networks (DNN) have achieved remarkable success in motion forecasting. However, most DNN-based methods suffer from catastrophic forgetting and fail to maintain their performance in previously learned scenarios after adapting to new data. Recent continual learning (CL) studies aim to mitigate this phenomenon by enhancing memory stability of DNN, i.e., the ability to retain learned knowledge. Yet, excessive emphasis on the memory stability often impairs learning plasticity, i.e., the capacity of DNN to acquire new information effectively. To address such stability-plasticity dilemma, this study proposes a novel CL method, synergetic memory rehearsal (SyReM), for DNN-based motion forecasting. SyReM maintains a compact memory buffer to represent learned knowledge. To ensure memory stability, it employs an inequality constraint that limits increments in the average loss over the memory buffer. Synergistically, a selective memory rehearsal mechanism is designed to enhance learning plasticity by selecting samples from the memory buffer that are most similar to recently observed data. This selection is based on an online-measured cosine similarity of loss gradients, ensuring targeted memory rehearsal. Since replayed samples originate from learned scenarios, this memory rehearsal mechanism avoids compromising memory stability. We validate SyReM under an online CL paradigm where training samples from diverse scenarios arrive as a one-pass stream. Experiments on 11 naturalistic driving datasets from INTERACTION demonstrate that, compared to non-CL and CL baselines, SyReM significantly mitigates catastrophic forgetting in past scenarios while improving forecasting accuracy in new ones. The implementation is publicly available at https://github.com/BIT-Jack/SyReM.

Problem

• Addresses catastrophic forgetting in motion forecasting DNNs
• Balances memory stability and learning plasticity trade-off
• Enables online continual learning from streaming data

Innovation

• Synergetic memory rehearsal for stability-plasticity balance
• Compact memory buffer with inequality constraint for stability
• Gradient-based selective replay mechanism enhancing plasticity