Breaking $\textit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning

2026-05-11

AI Summary

This work addresses the exploration collapse and insufficient solution diversity in existing population-based reinforcement learning methods for large language model (LLM) reasoning optimization, which often stem from “winner-takes-all” dynamics. To overcome this limitation, the authors propose Group Cooperative Policy Optimization (GCPO), a novel framework that shifts the training paradigm from individual competition to team-based collaboration. GCPO employs a determinantal point process (DPP)-inspired coverage metric to evaluate non-redundant, correct reasoning paths and allocates team-level rewards based on marginal contribution advantages. By introducing cooperative mechanisms in place of conventional competitive strategies—combined with reward-weighted semantic embeddings and cooperative policy gradient updates—the method significantly enhances both accuracy and diversity of solutions across multiple reasoning benchmarks, outperforming current reinforcement learning for verifiable reasoning (RLVR) approaches.

Abstract

Reinforcement learning with verifiers (RLVR) has become a central paradigm for improving LLM reasoning, yet popular group-based optimization algorithms like GRPO often suffer from exploration collapse, where the models prematurely converge on a narrow set of high-scoring patterns, lacking the ability to explore new solutions. Recent efforts attempt to alleviate this by adding entropy regularization or diversity bonus. However, these approaches do not change the \textit{winner-takes-all} nature, where rollouts still compete for individual advantage rather than cooperating for maximizing global diversity. In this work, we propose Group Cooperative Policy Optimization (GCPO), which shifts the training paradigm from rollout competition to team cooperation. Specifically, GCPO replaces independent rollout scoring with team-level credit assignment: a rollout is rewarded by how much it contributes to the team's valid solution coverage, rather than its individual accuracy. This coverage is described as a determinant volume over reward-weighted semantic embeddings, where only correct and non-redundant rollouts contribute to this volume. During advantage estimation, GCPO redistributes the collective team reward to each single rollout according to its average marginal contribution to the team. This cooperative training paradigm routes optimization toward non-redundant correct reasoning paths. Experiments across multiple reasoning benchmarks demonstrate that GCPO significantly improves both reasoning accuracy and solution diversity over existing approaches. Code will be released at $\href{https://github.com/bradybuddiemarch/gcpo}{this}$.

Problem

• exploration collapse
• winner-takes-all
• reasoning diversity
• cooperative policy optimization
• LLM reasoning

Innovation

• cooperative policy optimization
• determinant-based diversity
• team credit assignment
• reasoning diversity
• RLVR