RLVP: Penalize the Path, Reward the Outcome

2026-07-08

AI Summary

This work addresses the challenge of learning in real-world settings where agents interact through costly and irreversible actions, and sparse outcome-based rewards alone are insufficient to enforce path constraints—such as avoiding repeated dialing or adhering to business hours—while suffering from poor sample efficiency. The authors propose a novel reinforcement learning paradigm that systematically integrates verifiable path-based penalties with outcome rewards. They introduce four design principles for effective penalty formulation, which collectively prevent agents from falling into unproductive action traps and mitigate the vanishing advantage signal problem in fully failed trajectories. By constructing dense penalty signals based on intra-trajectory variance and incorporating a behavioral constraint verification mechanism, the method achieves high task success rates while reducing constraint violations to near zero, significantly outperforming baselines that rely solely on outcome rewards.

Abstract

Agents acting on our behalf in the real world (e.g. placing phone calls) must learn online from costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on the path, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples. Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiable penalty on the path meets this condition reliably, while a progress potential helps only where partial progress is reachable. The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of the inaction trap that arises when a penalty is used in isolation.

Problem

• real-world reinforcement learning
• path constraints
• verifiable penalties
• sample efficiency
• outcome-neutral constraints

Innovation

• path penalty
• verifiable rewards
• constraint-aware RL
• sample-efficient learning
• outcome-neutral constraints