Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation

2026-06-22

AI Summary

This work addresses the inefficiency and susceptibility to semantically plausible yet causally incorrect reward components in existing large language model–driven automatic reward design, which relies heavily on task-specific environmental feedback. To overcome these limitations, the authors propose the Causal Reward World Model (CRWM), which introduces causal structure modeling into reward design for the first time. CRWM constructs a task-agnostic, interpretable causal skeleton through offline pretraining and integrates a coarse-to-fine pretraining strategy with explicit mechanism disentanglement and a confidence-aware soft fusion module, enabling zero-shot reward generation without any environmental feedback. Evaluated on complex continuous control tasks, CRWM substantially reduces the latency of novel skill design, matches or surpasses state-of-the-art methods in performance, and demonstrates strong cross-task and cross-platform generalization capabilities.

Abstract

Automated Reward Design (ARD) aims to replace manual reward engineering in reinforcement learning with language-driven reward function synthesis. However, existing approaches based on large language models (LLMs) remain inherently correlation-driven, relying on iterative environmental feedback to refine reward hypotheses for each specific task. This paradigm not only results in inefficient reasoning but also makes LLMs susceptible to semantically plausible yet causally spurious reward components, leading to ineffective optimization. To address these limitations, we propose the Causal Reward World Model (CRWM), which explicitly models the causal topological relationships between candidate reward components and task-targeted physical variables through offline pre-training on multi-task interaction data. Based on a coarse-to-fine pre-training strategy, we introduce a joint optimization module that integrates Explicit Mechanism Decoupling with Confidence-Aware Soft Fusion to refine coarse structural priors using micro-level trajectories, thereby constructing a robust and interpretable causal skeleton. During inference, LLMs leverage CRWM as a task-irrelevant causal prior to constrain the reward generation, enabling zero-shot reward function design. Our work opens up a new white-box paradigm for the ARD problem. Extensive experiments on complex continuous control benchmarks demonstrate that CRWM generates executable reward functions without feedback-driven reward refinement, significantly reducing the design latency for acquiring new robotic skills while matching or surpassing state-of-the-art performance, and further exhibits strong generalization capabilities across unseen tasks and diverse robotic embodiments.

Problem

• Automated Reward Design
• Causal Reasoning
• Reinforcement Learning
• Reward Function Synthesis
• Zero-shot Learning

Innovation

• Causal Reward World Model
• Zero-shot Reward Design
• Automated Reward Design
• Causal Inference
• Language-driven RL