RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses

2026-04-30

AI Summary

This work addresses a critical oversight in existing approaches that deploy large language model (LLM)-generated reward functions for reinforcement learning: their reliability depends on both the current policy’s capability and the training stage. The paper formulates reward generation and deployment as a coupled problem and introduces a novel mechanism that jointly incorporates capability-aware validation and phase-aware deployment. Specifically, it establishes thresholds based on policy capability assessment and dynamically selects and validates a small set of candidate reward hypotheses according to the training phase, employing short-horizon forked validation and adaptive scheduling. Evaluated on sparse-reward manipulation tasks, the method significantly improves peak performance and training stability. Experiments further reveal that reward ranking becomes effective only after the policy surpasses a task-specific capability threshold, and no universal warm-up scheduling strategy exists.

Abstract

Large language models (LLMs) make reward design in reinforcement learning substantially more scalable, but generated rewards are not automatically reliable training objectives. Existing work has focused primarily on generating, evolving, or selecting reward candidates, while paying less attention to when such candidates can be verified and deployed during policy optimization. We study this deployment-time problem by treating generated rewards as reward hypotheses whose utility depends on the competence of the current policy and the phase of training. We propose \textsc{RHyVE}, a competence-aware verification and phase-aware deployment protocol that compares small sets of reward hypotheses from shared policy checkpoints using short-horizon fork verification. Our experiments show that reward rankings are unreliable at low competence but become informative after task-dependent thresholds. On a sparse manipulation task, phase-aware deployment improves peak and retained performance under a locked protocol. Updated LLM-generated reward-candidate experiments show candidate-family-dependent behavior: generated pools can exhibit phase-dependent winner changes, but no fixed warm-up schedule is universally optimal. Held-out schedule selection, conservative selector baselines, compute-matched controls, and scale controls further show that \textsc{RHyVE} is best understood as a verification-informed deployment protocol rather than a universal scheduler. Dense and all-failure boundary experiments delimit the scope of the method. Together, these results suggest that reward generation and reward deployment should be studied as coupled problems: generated rewards must be verified and deployed under changing policy competence.

Problem

• reward hypothesis
• competence-aware verification
• phase-aware deployment
• LLM-generated rewards
• policy optimization

Innovation

• reward hypothesis
• competence-aware verification
• phase-aware deployment
• LLM-generated rewards
• fork verification