Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models

2026-05-25

AI Summary

This work addresses the challenge of inconsistent performance improvement in large reasoning models under multi-domain reinforcement learning, where cross-domain interference often hinders effective policy optimization. To mitigate this issue, the authors propose Multi-domain Contrastive Policy Optimization (MCPO), which introduces contrastive learning into multi-domain policy optimization for the first time. MCPO constructs transferable reasoning trajectories as positive samples and erroneous trajectories as negative samples, enabling both cross-domain knowledge sharing and intra-domain knowledge consolidation. By aligning trajectory representations across domains into a unified semantic space, the method effectively suppresses detrimental interference while promoting beneficial knowledge transfer. Experiments demonstrate that MCPO significantly enhances model performance across multiple reasoning tasks, with certain scenarios even surpassing the performance of single-domain training.

Abstract

Post-training has significantly enhanced the reasoning capability of Large Reasoning Models (LRMs), especially with Reinforcement Learning (RL) like Group Relative Policy Optimization (GRPO). However, GRPO-style RL methods in multi-domain settings often fail to achieve consistent improvements across all domains due to inherent interference in policy optimization. Prior studies on multi-domain RL primarily focus on alleviating cross-domain interference, while often neglecting the pivotal role of knowledge sharing, which we argue is the key to transforming cross-domain interactions from harmful competition into beneficial transfer. To address this limitation, we propose Multi-domain Contrastive Policy Optimization (MCPO), which analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing and in-domain knowledge consolidation in a contrastive manner. Specifically, for a given prompt, MCPO identifies transferable reasoning trajectories from other domains as positive examples, while treating incorrect rollouts as negative ones. It then encourages consistent representations for positive pairs and pushes negative pairs apart, thereby facilitating knowledge transfer and reducing interference. Moreover, MCPO aligns intra-domain correct rollouts to build a consolidated representation space. In this way, MCPO contrastively learns a harmonious representation space that can accommodate diverse multi-domain knowledge. Empirical results show that MCPO improves the reasoning capabilities of LRMs across multiple domains and even outperforms single-domain training in some cases. Code is available at https://github.com/Maricalce/MCPO.

Problem

• multi-domain reinforcement learning
• cross-domain interference
• knowledge sharing
• large reasoning models
• policy optimization

Innovation

• Multi-domain Reinforcement Learning
• Contrastive Learning
• Knowledge Transfer
• Policy Optimization
• Large Reasoning Models