Tandem Reinforcement Learning with Verifiable Rewards

2026-06-26

AI Summary

This work addresses the challenge that reasoning chains generated by strong agents in reinforcement learning are often difficult for weaker agents or humans to understand due to linguistic inconsistency and poor readability. To mitigate this, the authors propose a two-agent collaborative training framework in which a strong agent and a frozen weak agent stochastically alternate in generating reasoning steps, jointly optimizing a verifiable reward shared at the team level. This approach explicitly encourages the strong agent to produce clear, followable reasoning paths aligned with the weak agent’s capabilities. The method introduces collaborative training into reinforcement learning with verifiable rewards (RLVR) for the first time, implemented via the GRPO algorithm on the Qwen3-4B-Instruct model. Experiments on competitive mathematics tasks demonstrate that the model maintains its original reasoning proficiency while significantly improving handoff robustness to weaker agents, mitigating distributional shift, and generating more interpretable reasoning chains.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly improved the reasoning capability of large language models, reaching expert or even superhuman performance in domains such as competition math. However, whether weaker agents and humans can actually harness this capability is far less certain, with RLVR documented to drift reasoning toward idiosyncratic patterns such as poor readability and language mixing. Tandem training is a recently introduced paradigm that targets this compatibility problem: a trained, stronger senior co-generates each rollout with a frozen, weaker junior, and the two are rewarded as a team, so the senior is pushed to reason in ways the junior can follow. Yet this paradigm has so far been demonstrated only in proof-of-concept settings, leaving open whether it scales to the long chains of thought of the modern RLVR pipeline. In this work, we propose Tandem Reinforcement Learning (TRL), which carries the tandem training paradigm into RLVR. In TRL, the senior and a frozen junior alternate stochastically to co-generate the reasoning, the resulting generation is rewarded, and the standard GRPO loss is applied to the senior. Training Qwen3-4B-Instruct on competition math, we find that TRL matches vanilla GRPO on solo reasoning capability while three properties emerge together from the same rollout structure: stronger handoff robustness with the junior, reduced distributional drift from the junior, and a chain-of-thought more legible to the junior. Our results demonstrate a promising route for RLVR with practical payoffs in multi-model communication and human compatibility.

Problem

• reinforcement learning with verifiable rewards
• tandem training
• reasoning compatibility
• distributional drift
• chain-of-thought legibility

Innovation

• Tandem Reinforcement Learning
• verifiable rewards
• chain-of-thought legibility
• distributional drift
• multi-model compatibility