Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

2026-09-16

AI Summary

研究通过仅优化约10万个偏置参数并使用多数投票伪标签作为奖励，实现了在不依赖标记数据的情况下提高模型推理能力，解决了测试时强化学习需要大量参数优化的问题。

Abstract

Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudo-labels as rewards and optimizes only approximately 100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy, slightly exceeding our own labeled bias-steering reproduction while optimizing 76,000x fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability. These results demonstrate that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.

Problem

• test-time reinforcement learning
• reward signal
• optimization space
• bias parameters
• label-free

Innovation

• test-time reinforcement learning
• bias-only optimization
• majority-vote pseudo-labels
• parameter efficiency