CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

2026-08-10

AI Summary

This work addresses the challenge of test-time reinforcement learning in the absence of ground-truth rewards, where conventional majority voting often overlooks correct but minority reasoning trajectories and fails to discern fine-grained consistency among them. To overcome these limitations, the authors propose CoRE, a method that constructs a graph over N reasoning trajectories by integrating answer consistency, reasoning similarity, and generation confidence. CoRE employs replicator dynamics to extract a dominant set, yielding calibrated hierarchical pseudo-labels and self-supervised rewards. Theoretically, CoRE strictly generalizes majority voting, recovering the correct answer even when it lies in the minority, with confidence calibration multiplicatively lowering the recovery threshold. Across 42 experiments spanning seven backbone models and five benchmarks, CoRE achieves an average gain of 21.7 points—surpassing majority voting’s 20.4—and leads by up to 7.5 points on contentious questions, while reaching baseline accuracy in 54%–70% fewer steps.

Abstract

On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over $N$ sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with \emph{CoRE} (Consensus Rewards via Equilibrium): the $N$ roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), \emph{CoRE} improves the untrained base by $+21.7$ points on average versus $+20.4$ for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to $+7.5$ points, and reaches the voting baseline's plateau accuracy in $54$--$70$\% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.

Problem

• test-time reinforcement learning
• reward design
• majority voting
• pseudo-labeling
• consensus

Innovation

• Consensus Rewards
• Test-Time Reinforcement Learning
• Replicator Dynamics
• Dominant Set
• Self-Supervised Reward