CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

2026-03-10

AI Summary

Existing reinforcement learning from verbal reward (RLVR) methods rely solely on final-answer correctness as the reward signal, neglecting the validity of intermediate reasoning steps. This oversight often leads to hallucination and answer copying, undermining model generalization and robustness. To address this limitation, this work introduces contrastive learning into RLVR policy optimization for the first time. By constructing a contrastive loss based on multiple successful reasoning trajectories, the approach encourages large language models to identify invariant structures within correct reasoning paths, thereby enabling fine-grained supervision over the entire reasoning process. Moving beyond single-path supervision, the proposed method effectively mitigates step-level reasoning inconsistencies and significantly outperforms existing RLVR baselines across multiple reasoning benchmarks, enhancing model reliability, generalization, and robustness.

Technology Category

• Machine Learning: Reinforcement Learning
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.

Problem

• Reinforcement Learning
• Reasoning
• Hallucination
• Generalization
• Robustness

Innovation

• Contrastive Learning
• Policy Optimization
• RLVR
• Reasoning Robustness
• LLM Generalization