reward function design

This paper addresses the misalignment between reward models and true objectives in deep reinforcement learning, as well as the resulting limitations in policy optimization. To this end, it introduces— for the first time—a unified taxonomy that systematically organizes reward modeling across three orthogonal dimensions: modeling source (explicit vs. implicit), mechanism design (supervised vs. interactive), and learning paradigm (static vs. dynamic). The survey comprehensively covers mainstream approaches—including inverse reinforcement learning, preference learning, language-model-based feedback, human demonstration distillation, contrastive learning, and online interactive modeling—and critically analyzes evaluation methodologies and practical deployment challenges. This work fills a critical gap in the literature by providing the first systematic, cross-cutting review of reward modeling. It clarifies the technical evolution of the field and identifies four key research frontiers: scalability, generalization, robustness, and human-AI alignment.

In reinforcement learning, reward function design faces critical challenges including delayed signals, ambiguity, misalignment with task objectives, and induction of undesirable behaviors. To address these, this paper proposes three novel reward mechanisms: teacher-driven, adaptive explainable, and agent-autonomous reward generation. Our core contributions are the first-ever adaptive explainable reward design method and a meta-learning–driven autonomous reward generation framework—enabling a paradigm shift from expert-guided reward specification to online inverse reward modeling by the agent. Technically, we integrate reward shaping, eXplainable AI (XAI)-informed reward modeling, policy-value alignment, and online inverse reward design. Experiments across multiple sparse-reward benchmarks demonstrate over 40% faster training convergence, significantly improved policy robustness, and high reward interpretability—validated by domain experts with 92% inter-rater agreement.

Sparse, delayed, and weakly informative reward signals severely hinder the efficiency of reinforcement learning, and existing reward shaping methods often fail to adapt to dynamic environments. This work proposes the first unified analytical framework encompassing temporal, informational, and theoretical dimensions to systematically categorize and compare twelve classes of dynamic reward shaping and related adaptive mechanisms. It clearly distinguishes between parameter corrections and state-dependent modifications, and precisely delineates the boundaries among additive shaping, reward replacement, and correlated guidance. Through integrated theoretical analysis and taxonomic synthesis, the study identifies conditions under which optimality is preserved in the presence of modern RL components such as experience replay, bootstrapped critics, and reward normalization, and for the first time elucidates the intrinsic relationship between adaptation rate and learning stability.

In reinforcement learning, Linear Temporal Logic (LTL) tasks suffer from sparse rewards, hindering subgoal guidance, slowing policy convergence, and degrading robustness. To address this, we propose a progress-aware adaptive reward shaping method: for the first time, we quantify LTL satisfaction progress as a continuous, differentiable reward signal and design an online mechanism to dynamically update the reward function according to the agent’s current learning state. Our approach integrates LTL task compilation, formal progress modeling, and deep RL frameworks (PPO/SAC). Evaluated across multiple benchmark environments, the method significantly accelerates convergence, increases average expected return by 23%, improves task completion rate by 31%, and outperforms both traditional sparse-reward baselines and handcrafted reward-shaping approaches in terms of robustness and generalization.

To address low sample efficiency and unstable convergence in reinforcement learning caused by sparse rewards, this paper proposes an adaptive reward shaping method grounded in historical success rates. The method models state-dependent success probability as a time-varying Beta distribution—explicitly capturing epistemic uncertainty for the first time in this context. It further introduces an uncertainty-driven stochastic annealing strategy that naturally balances exploration and exploitation. For scalable, model-free, nonparametric success-rate estimation in high-dimensional continuous state spaces, the approach integrates kernel density estimation (KDE) with Random Fourier Features. Experiments demonstrate substantial improvements in sample efficiency and convergence stability on extremely sparse-reward tasks, consistently outperforming state-of-the-art reward shaping and intrinsic motivation baselines across diverse benchmarks.

For implicit-standard tasks—such as autonomous driving, humanoid locomotion, and dexterous manipulation—where “desirable behavior” is difficult to formalize and reward design suffers from subjectivity, this paper proposes REvolve: the first end-to-end reward evolution framework. REvolve integrates large language model (LLM)-driven reasoning, structured human feedback modeling, evolutionary algorithms, and deep reinforcement learning to automatically transform qualitative judgments into quantitative reward functions. Through iterative closed-loop cycles—comprising reward generation, pairwise ranking, mutation, and policy evaluation—it dynamically incorporates expert implicit knowledge into reward optimization. Evaluated on three canonical tasks, REvolve-generated reward functions yield policies with significantly improved performance, consistently outperforming state-of-the-art baselines including rule-based, inverse reinforcement learning, and zero-shot LLM-derived rewards.

This work addresses the challenge in reinforcement learning that handcrafted reward functions are time-consuming to design and often misaligned with task objectives. To this end, the paper introduces the Trajectory Alignment Coefficient (TAC) as a metric quantifying the consistency between a reward function and expert preferences, and for the first time employs TAC both as an auxiliary signal for reward tuning and as a direct learning objective for reward modeling. Furthermore, the authors develop Soft-TAC, a differentiable approximation of TAC, enabling end-to-end learning of reward models from human preference data. Experiments demonstrate that in Lunar Lander, TAC significantly improves reward function performance while reducing the cognitive burden of manual tuning; in Gran Turismo 7, reward models trained with Soft-TAC yield more diverse and effective behavioral policies compared to standard cross-entropy approaches.

Reward function design in reinforcement learning heavily relies on human expertise, resulting in poor generalizability and high engineering costs. Method: We propose the first fully autonomous framework for reward function generation and optimization—requiring no predefined evaluation metrics, environment source code, or human feedback. Leveraging large language models (LLMs), it integrates task-semantic parsing with multi-round sampling to enable model-agnostic, unsupervised generation, execution, and evaluation of reward functions. The LLM autonomously infers task-specific performance metrics and selects high-performing reward functions. Contribution/Results: Experiments across multiple control benchmarks demonstrate that our approach matches or surpasses state-of-the-art methods (e.g., EUREKA). Notably, it achieves competitive performance even with low-cost LLMs, substantially reducing human intervention while improving generality and automation in reward design.

Designing reward functions for reinforcement learning (RL) agents in games traditionally relies heavily on domain expertise and struggles to adapt to dynamic content changes. Method: This paper proposes an LLM-based automated iterative reward weight optimization method that takes user-specified behavioral objectives as input and leverages agent training feedback—such as success rate and episode length—to perform closed-loop, multi-round LLM reasoning for reward weight self-calibration, eliminating manual intervention. Contribution/Results: To our knowledge, this is the first work to integrate LLMs into online adaptive optimization of RL reward functions, substantially reducing dependence on human experts. Evaluated on a racing task, the approach improves agent success rate from 9% to 80% and reduces average lap steps to 855—performance approaching that achieved by expert manual tuning.

This work addresses the challenge of sparse rewards in reinforcement learning, which hinders effective exploration, and the risk of reward gaming associated with handcrafted reward shaping. The authors propose the first integration of vision-language models (VLMs) into potential-based reward shaping (PBRS), leveraging a lightweight VLM to automatically learn a potential function by evaluating preferences over pairs of state images. This approach preserves the original optimal policy while eliminating human-induced design bias. Notably, the method requires only a small-scale VLM to efficiently generate preference labels, substantially improving sample efficiency. Empirical results in Meta-World and Franka Kitchen environments demonstrate strong robustness against reward gaming, confirming that even low-accuracy VLMs can effectively accelerate learning.

In multi-objective reinforcement learning (MORL), manually designed reward functions suffer from subjectivity and poor generalizability. To address this, we propose MORSE—a framework that automatically synthesizes multiple human-specified heuristic rewards into a unified, differentiable, and optimizable composite reward function via a bilevel optimization mechanism. Crucially, MORSE introduces exploratory noise derived from both task performance and prediction errors of a random neural network, enhancing policy exploration and mitigating local optima. Integrated with policy gradient optimization, MORSE is evaluated across diverse robot control tasks in MuJoCo and Isaac Sim. Results demonstrate that it achieves or surpasses the performance of hand-tuned reward functions in both Pareto optimality and overall task performance, while significantly reducing reward engineering effort.