Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

2026-07-16

AI Summary

This work addresses a key limitation in existing reinforcement learning approaches that rely on entropy for advantage shaping: their inability to distinguish between beneficial uncertainty and detrimental confusion, which compromises the accurate reflection of generation correctness. To overcome this, the paper proposes Contrastive Policy Optimization (CPO), which introduces token-level contrastive divergence as a correctness-aware advantage signal for the first time. CPO constructs a more precise policy update mechanism by contrasting the reference model–guided distribution against the original generation distribution. The method unifies on-policy distillation as a special case and effectively mitigates the zero-advantage problem. Experiments demonstrate that CPO significantly outperforms entropy-based RLVR methods on both in-domain and out-of-domain benchmarks, maintaining strong generalization while revealing an optimization dynamic wherein correct responses promote exploration and incorrect ones drive exploitation.

Abstract

Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.

Problem

• reinforcement learning with verifiable rewards
• advantage shaping
• entropy
• correctness signal
• token-level correctness

Innovation

• Contrastive Policy Optimization
• correctness-aware advantage shaping
• token-level contrastive disagreement
• reinforcement learning with verifiable rewards
• on-policy distillation