Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models

2026-05-08

AI Summary

This work addresses the challenge of ineffective experience sharing among heterogeneous large language models during post-reinforcement learning training, stemming from incompatibilities in parameters, objectives, and tokenizers. To overcome this, the authors propose a mutual reinforcement learning framework that enables structured experience exchange across diverse model architectures for the first time. The framework integrates three core components: Shared Experience Exchange (SEE), Multi-Worker Resource Allocation (MWRA), and Tokenizer-Heterogeneous Layer (THL), complemented by novel mechanisms including Peer Rollout Pooling, XGRPO, and Success-Gated Transfer (SGT). Experimental results demonstrate that SGT achieves an optimal trade-off between stability and supportiveness, significantly enhancing collaborative training performance among heterogeneous models.

Abstract

We introduce Mutual Reinforcement Learning, a framework for concurrent RL post-training in which heterogeneous LLM policies exchange typed experience while keeping separate parameters, objectives, and tokenizers. The framework combines a Shared Experience Exchange (SEE), Multi-Worker Resource Allocation (MWRA), and a Tokenizer Heterogeneity Layer (THL) that retokenizes text and aligns token-level traces across incompatible vocabularies. This substrate makes the experience-sharing design question operational across model families. We instantiate three controlled probes on top of GRPO: data-level rollout sharing via Peer Rollout Pooling (PRP), value-level advantage sharing via Cross-Policy GRPO Advantage Sharing (XGRPO), and outcome-level success transfer via Success-Gated Transfer (SGT). A contextual-bandit analysis characterizes their structural positions on a stability-support trade-off: PRP pays density-ratio variance and THL residual costs, XGRPO preserves on-policy actor support while changing scalar baselines, and SGT supplies a rescue-set score direction toward verified peer successes. In the evaluated regime, outcome-level sharing occupies the favorable point of this trade-off.

Problem

• Mutual Reinforcement Learning
• Heterogeneous Language Models
• Experience Sharing
• Tokenizer Heterogeneity
• Reinforcement Learning Post-training

Innovation

• Mutual Reinforcement Learning
• Tokenizer Heterogeneity Layer
• Experience Sharing
• Heterogeneous Language Models
• Success-Gated Transfer