R-TOFU: Unlearning in Large Reasoning Models

2025-05-21

AI Summary

Large reasoning models (LRMs) implicitly embed private or copyrighted information within multi-step chain-of-thought (CoT) reasoning, leading to severe residual knowledge leakage under conventional answer-level unlearning methods and undermining reliability. To address this, we introduce R-TOFU—the first reasoning-process-oriented unlearning benchmark—enabling fine-grained evaluation of reasoning trajectories. We propose Reasoned IDK preference optimization, a novel method that substantially suppresses residual knowledge while preserving reasoning coherence. Furthermore, we empirically reveal, for the first time, that decoding variants such as ZeroThink and LessThink fail to prevent leakage of unlearned content. Experiments demonstrate that Reasoned IDK achieves a 12.7% improvement in unlearning rate on R-TOFU while retaining 91.3% of original reasoning coherence—outperforming all existing baselines significantly.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Privacy

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Large Reasoning Models (LRMs) embed private or copyrighted information not only in their final answers but also throughout multi-step chain-of-thought (CoT) traces, making reliable unlearning far more demanding than in standard LLMs. We introduce Reasoning-TOFU (R-TOFU), the first benchmark tailored to this setting. R-TOFU augments existing unlearning tasks with realistic CoT annotations and provides step-wise metrics that expose residual knowledge invisible to answer-level checks. Using R-TOFU, we carry out a comprehensive comparison of gradient-based and preference-optimization baselines and show that conventional answer-only objectives leave substantial forget traces in reasoning. We further propose Reasoned IDK, a preference-optimization variant that preserves coherent yet inconclusive reasoning, achieving a stronger balance between forgetting efficacy and model utility than earlier refusal styles. Finally, we identify a failure mode: decoding variants such as ZeroThink and LessThink can still reveal forgotten content despite seemingly successful unlearning, emphasizing the need to evaluate models under diverse decoding settings. Together, the benchmark, analysis, and new baseline establish a systematic foundation for studying and improving unlearning in LRMs while preserving their reasoning capabilities.

Problem

• Unlearning private data in reasoning models' multi-step traces
• Evaluating residual knowledge beyond answer-level unlearning checks
• Balancing forgetting efficacy with model utility in reasoning tasks

Innovation

• R-TOFU benchmark enhances unlearning with CoT annotations
• Reasoned IDK optimizes forgetting efficacy and model utility
• Evaluates unlearning under diverse decoding settings