CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization

2026-05-19

AI Summary

This work addresses a key limitation in reward-verifiable reinforcement learning (RLVR), where existing methods assign uniform rewards to all generated tokens, failing to distinguish critical reasoning steps from irrelevant content. To resolve this, the authors propose Contrastive Evidence Policy Optimization (CEPO), which innovatively leverages incorrect answers as contrastive signals: positive credit is allocated only when the policy explicitly favors correct answers while rejecting incorrect ones, thereby precisely attributing reward to decisive reasoning steps. CEPO integrates rejection-sampled incorrect answers, policy gradient optimization, and a self-distillation framework to enable contrastive credit assignment without additional sampling overhead or information leakage, while preserving safety guarantees. Experimental results demonstrate that CEPO significantly outperforms baselines such as GRPO across five mathematical reasoning benchmarks, achieving average accuracies of 43.43% and 60.56% on 2B and 4B models, respectively.

Abstract

When a model produces a correct solution under reinforcement learning with verifiable rewards (RLVR), every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler. A natural fix is to condition the model on the correct answer as a teacher, identifying tokens it would have generated differently had it known the answer. Prior work shows this either corrupts training by leaking the answer into the gradient, or produces a weak signal that cannot distinguish decisive steps from filler, since both look equally surprising relative to the model's baseline. We propose Contrastive Evidence Policy Optimization (CEPO), which asks a sharper question at every token: not just "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?" A token satisfying both is a genuine reasoning step; one satisfying neither is filler. The wrong-answer teacher is constructed from rejected rollouts already in the training batch, incurring no additional sampling cost. We prove CEPO inherits all structural safety guarantees of the prior state of the art while strictly sharpening credit at decisive tokens, with the improvement vanishing exactly at filler positions. Empirically, CEPO achieves 43.43% and 60.56% average accuracy across five multimodal mathematical reasoning benchmarks at 2B and 4B scale, respectively, versus 41.17% and 57.43% for GRPO under identical training budgets. Distribution-matching self-distillation methods (OPSD, SDPO) fall below the untrained baseline, empirically confirming the information leakage our theory predicts. Our code is available at https://github.com/ahmedheakl/CEPO.

Problem

• reinforcement learning with verifiable rewards
• token-level credit assignment
• reasoning steps
• reward signal
• self-distillation

Innovation

• Contrastive Evidence Policy Optimization
• RLVR
• self-distillation
• credit assignment
• reasoning steps