Mechanism-Guided Selective Unlearning for RLVR-Induced Reasoning

2026-06-17

AI Summary

This work addresses the challenge in reinforcement learning with verifiable rewards (RLVR) where models struggle to precisely unlearn specific reasoning capabilities without degrading retained knowledge. The authors propose MAST, a mechanism-alignment-based selective update method that identifies and fine-tunes only the subset of attention projection subspaces most relevant to the target forgetting task. This selection leverages three criteria: deviation from principal component energy, magnitude of parameter updates, and coupling strength with forgetting gradients. Combining attention-based selective gradient updates with NPO/SimNPO optimization objectives on Qwen2.5-Math and Qwen3, MAST achieves effective定向 forgetting. Experiments show that on Qwen2.5-Math-1.5B, accuracy on the MATH forget set drops significantly from 45/150 to 37/150 (p=0.0078), while GSM8K performance improves by 0.8% and the MATH retain set declines by only 0.5%, with consistent results across multiple models and random seeds.

Abstract

We propose MAST (Mechanism-Aligned Selective Targeting), a mechanism-guided method for unlearning RLVR-induced reasoning with substantially lower collateral damage than standard full-parameter updates. In matched SFT/RLVR checkpoints on Qwen2.5-Math-1.5B and Qwen3-1.7B-Base, the SFT-to-RLVR increment differs sharply from the SFT update in token-level delta-log-probability, and full-parameter gradient ascent forgets only by damaging retain MATH and GSM8K. MAST ranks attention-projection tensors by off-principal energy, update magnitude, and forget-gradient coupling magnitude, then updates only the top-ranked subset. On the primary model, MAST induces statistically significant target forgetting (MATH forget 45/150 to 37/150; McNemar p=0.0078) while preserving GSM8K (+0.8 pp) and MATH retain (-0.5 pp). The advantage reproduces across seeds, NPO/SimNPO objectives, and Qwen3, where MAST preserves GSM8K while full-parameter unlearning collapses it.

Problem

• selective unlearning
• RLVR-induced reasoning
• collateral damage
• mechanism-guided
• target forgetting

Innovation

• selective unlearning
• mechanism-guided
• RLVR-induced reasoning
• attention-projection tensors
• collateral damage minimization