Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

2026-06-13

AI Summary

Existing reinforcement learning–based unlearning methods for large language models suffer from inefficiency due to repeated sampling of already-converged, easy examples, thereby neglecting challenging samples near decision boundaries. This work proposes ReRULE, the first approach to integrate off-policy replay into the unlearning process: difficult samples with low rewards are stored in a replay buffer early in training and later reused via importance sampling to focus optimization on critical boundary cases. Theoretical analysis demonstrates that ReRULE achieves a tighter convergence bound for hard examples compared to purely on-policy methods. Empirical results on the MUSE-Books benchmark show that ReRULE improves retention quality from 46.3 to 56.2 with only a 5–11% increase in training overhead, yielding particularly pronounced gains when the distinction between hard and easy samples is substantial.

Abstract

LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility. Recent RL-based methods such as RULE reformulate unlearning as learning a refusal behavior, but their on-policy optimization repeatedly samples from the same forget and retain/boundary prompts throughout training. We identify a critical inefficiency in this process: easy cases quickly converge and provide little useful gradient signal, while hard cases near the forget/retain boundary continue to produce low-reward rollouts that are discarded after a single use. To address this issue, we propose ReRULE, an off-policy replay enhancement for reinforcement unlearning. ReRULE stores low-reward hard-case rollout groups in a replay buffer during early GRPO training and reuses them in later stages through importance-sampled off-policy updates, redirecting computation toward boundary cases that still require learning. Theoretically, we show that ReRULE yields a tighter hard-case convergence bound than pure on-policy RULE. Empirically, ReRULE improves MUSE-Books Retain Quality from 46.3 to 56.2 while adding only 5--11% training time across benchmarks. Its limited improvement on the simpler TOFU setting further supports the intended conditional behavior: replay is most beneficial when the hard/easy disparity is pronounced.

Problem

• LLM unlearning
• reinforcement learning
• off-policy replay
• sample efficiency
• forget/retain boundary

Innovation

• off-policy replay
• reinforcement unlearning
• hard-case sampling
• importance sampling
• LLM unlearning