Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

2026-06-24

AI Summary

This work addresses the challenge of scaling reward modeling in long-horizon agent interactions, where irreversible actions, stochastic feedback, and high annotation costs hinder progress. The authors propose “progress advantage”—a step-level advantage signal that requires no additional training and emerges naturally as a byproduct of standard reinforcement learning (RL) fine-tuning. It is derived directly from the log-probability ratio between the current policy and a reference policy, is domain-agnostic, annotation-free, and grounded theoretically in stochastic Markov decision processes. Experiments across five benchmarks and four model families demonstrate that progress advantage consistently outperforms confidence-based baselines in test-time scaling, uncertainty quantification, and failure attribution, often matching or surpassing specialized reward models trained explicitly for these tasks.

Abstract

Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.

Problem

• process reward models
• agentic settings
• long-horizon interactions
• stochastic environment feedback
• step-level evaluation

Innovation

• progress advantage
• reward-free evaluation
• reinforcement learning post-training
• step-level scoring
• LLM agents