Reward as An Agent for Embodied World Models

2026-06-18

AI Summary

This work addresses critical limitations in existing reinforcement learning approaches, which rely on conservative trajectory replay that restricts exploration capacity and behavioral diversity while remaining vulnerable to reward hacking. To overcome these issues, the authors propose a “Reward-as-Agent” framework that actively validates the reliability of agent behaviors and integrates a novel DynDiff-GRPO algorithm to explicitly expand action-space exploration. This enables dynamic, perception-aware diverse replay, breaking away from traditional conservative replay paradigms. Evaluated across multiple open-source world models, the method significantly improves model accuracy and effectively mitigates reward hacking, thereby demonstrating the feasibility and robustness of large-scale exploration grounded in reliable validation mechanisms.

Abstract

While RL has become a promising tool for refining world models, existing methods largely rely on conservative rollouts near the training distribution, limiting exploration, behavioral diversity, and richer dynamic discovery. In this work, we challenge this conservative paradigm. We argue that the core limitation is not exploration itself, but the lack of reliable verification strategies to support broader exploration. Without reliable verification, expanded exploration becomes highly susceptible to reward hacking, where policies exploit imperfect rewards without achieving genuine improvement. To evaluate this motivation, we instantiate our method in embodied world models, where physical plausibility, and task completion provide a rigorous testbed for scalable RL under complex dynamics. On the verification side, we introduce Reward as an Agent, an agentic reward framework that actively evaluates generated behaviors to provide robust reward signals and mitigate reward hacking under distribution shifts. On the exploration side, we introduce Dynamic-Aware Rollout Diversification through DynDiff-GRPO, which explicitly expands action-space exploration to diversify trajectories, broaden state-action coverage, and encourage richer embodied behaviors beyond conservative rollout regimes. By unifying Reward as an Agent with DynDiff-GRPO, we enable RL on a more reliable reward foundation with substantially diversified sampling, effectively mitigating reward hacking while yielding significant accuracy gains across multiple open-source world models, thereby demonstrating that broader exploration can scale successfully when grounded in robust verification.

Problem

• reward hacking
• world models
• exploration
• verification
• embodied AI

Innovation

• Reward as an Agent
• reward hacking
• embodied world models
• DynDiff-GRPO
• dynamic-aware exploration