RL Forgets! Towards Continual Policy Optimization

2026-07-05

AI Summary

This work addresses the severe catastrophic forgetting problem that arises when applying reinforcement learning to multimodal continual post-training. To this end, the authors introduce MRCL, the first diverse benchmark for multimodal reasoning in continual learning, and propose CPO, a replay-free continual policy optimization framework. CPO mitigates policy drift by integrating KL divergence regularization toward task-prior behaviors with parameter movement constraints. Experiments demonstrate that CPO reduces forgetting by 13.7% and enhances pre-trained capabilities by 7.0% on Qwen3-VL-8B, with consistent improvements across various model scales. These results establish both theoretical grounding and a practical solution for replay-free multimodal continual reinforcement learning.

Abstract

Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks. Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting. However, the belief remains insufficiently validated, as existing evidence is largely drawn from outdated or homogeneous benchmarks. To revisit this assumption, we introduce MRCL, a Multimodal Reasoning Continual Learning benchmark built from diverse and recently released multimodal datasets. Experiments on MRCL show that reinforcement learning can still suffer from severe catastrophic forgetting during continual post-training. To address this challenge, we propose Continual Policy Optimization (CPO), a replay-free framework grounded in the prior-task behavioral KL objective. CPO uses a theoretically justified parameter-movement regularization to limit policy drift on previous tasks. Extensive experiments across multiple model scales demonstrate that CPO consistently reduces forgetting while preserving, and in some cases improving, pretrained model capabilities. On Qwen3-VL-8B, CPO reduces forgetting by 13.7\% and improves pretrained capability by 7.0\%. The implementation code is available at https://github.com/MaolinLuo/CPO.

Problem

• catastrophic forgetting
• reinforcement learning
• continual learning
• multimodal reasoning
• policy optimization

Innovation

• Continual Policy Optimization
• catastrophic forgetting
• reinforcement learning
• parameter-movement regularization
• multimodal continual learning