Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching

2026-09-07

AI Summary

本文针对长周期任务中强化学习训练语言模型效率低的问题，提出了一种名为逐步点匹配的无偏密集奖励方法，通过奖励部分进展来提高学习效率。

Abstract

Current paradigms for training language models via reinforcement learning rely heavily on sparse outcome rewards. However, as we pursue tasks that require longer and more complicated trajectories, such strategies result in slow learning. Prior work has attempted to address this problem by rewarding partial progress; however, naive formulations are often biased and converge to suboptimal policies. We show that a simple and unbiased dense reward formulation, which we term progressive point matching, scales exponentially more efficiently to long-horizon tasks by rewarding partial progress on a segment level, both theoretically and empirically via synthetic environments. We then show how progressive point matching can be practically instantiated using a single reference trajectory per task. On extremely hard math reasoning problems, sparse outcome rewards cannot make any progress, whereas segment-level rewards enable improvements at larger test-time token budgets when measured by success rate or pass@k.

Problem

• reinforcement learning
• language models
• sparse rewards
• long-horizon tasks
• partial progress

Innovation

• progressive point matching
• dense reward
• long-horizon tasks
• partial progress