Tail-Likelihood Reinforcement Learning

2026-09-02

AI Summary

该研究提出Tail-Likelihood强化学习方法，直接优化获得高回报事件的概率，以解决传统强化学习中平均回报隐藏的稀有但高回报样本被忽略的问题。

Abstract

Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-(k) gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.

Problem

• reinforcement learning
• average reward
• rare high-reward rollouts
• probability mass
• high-reward outcomes

Innovation

• Tail-Likelihood Reinforcement Learning
• High-Reward Rollouts
• Reward Thresholds
• Advantage Function Modification