CiPO: Counterfactual Unlearning for Large Reasoning Models through Iterative Preference Optimization

2026-04-17

AI Summary

This work addresses the challenge of effectively removing specific knowledge from large reasoning models without compromising their complex reasoning capabilities. Existing approaches struggle to fully eliminate targeted information from the model’s chain-of-thought while preserving its inferential competence. To overcome this limitation, the authors propose a counterfactual unlearning framework based on iterative preference optimization, which formalizes forgetting as a directed intervention on the reasoning process. The method generates logically coherent yet factually counterfactual reasoning paths that contradict the target knowledge and uses them to construct preference pairs for iterative fine-tuning. Experiments across multiple benchmarks demonstrate that this approach successfully eradicates the target knowledge from both intermediate reasoning steps and final outputs, while substantially retaining the model’s advanced reasoning performance.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Machine unlearning has gained increasing attention in recent years, as a promising technique to selectively remove unwanted privacy or copyrighted information from Large Language Models that are trained on a massive scale of human data. However, the emergence of Large Reasoning Models (LRMs), which emphasize long chain-of-thought (CoT) reasoning to address complex questions, presents a dilemma to unlearning: existing methods either struggle to completely eliminate undesired knowledge from the CoT traces or degrade the reasoning performances due to the interference with the reasoning process. To this end, we introduce Counterfactual Unlearning through iterative Preference Optimization (CiPO), a novel framework that redefines unlearning as the targeted intervention of the CoT reasoning in LRMs. More specifically, given a desired unlearning target answer, CiPO instructs LRMs to generate a logically valid counterfactual reasoning trace for preference tuning. As the LRM adjusts to the counterfactual trace, CiPO iteratively updates the preference learning data to increase the discrepancy from the original model. This iterative loop ensures both desirable unlearning and smooth optimization, effectively mitigating the dilemma. Experiments on challenging benchmarks demonstrate that CiPO excels at unlearning, completely removing knowledge from both the intermediate CoT steps and the final answer, while preserving the reasoning abilities of LRMs.

Problem

• machine unlearning
• Large Reasoning Models
• chain-of-thought reasoning
• knowledge removal
• reasoning preservation

Innovation

• Counterfactual Unlearning
• Iterative Preference Optimization
• Large Reasoning Models
• Chain-of-Thought Reasoning
• Machine Unlearning