Curiosity-Critic: Cumulative Prediction Error Improvement as a Tractable Intrinsic Reward for World Model Training

2026-04-20

AI Summary

This work addresses a key limitation in traditional curiosity-driven exploration methods, which rely solely on the prediction error of the current transition and overlook the cumulative improvement in predictive accuracy across all previously visited transitions. To overcome this, the authors propose Curiosity-Critic, a novel approach that introduces the improvement in cumulative prediction error as an intrinsic reward. By jointly training a world model and a scalar Critic network, the method estimates an asymptotic error baseline online for each state transition, thereby distinguishing reducible (epistemic) from irreducible (aleatoric) uncertainty without requiring prior knowledge of a noise floor. Evaluated in stochastic grid-world environments, Curiosity-Critic significantly accelerates convergence and enhances the final accuracy of the world model, outperforming established baselines such as prediction-error-based and visitation-count-based intrinsic rewards.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Local prediction-error-based curiosity rewards focus on the current transition without considering the world model's cumulative prediction error across all visited transitions. We introduce Curiosity-Critic, which grounds its intrinsic reward in the improvement of this cumulative objective, and show that it reduces to a tractable per-step form: the difference between the current prediction error and the asymptotic error baseline of the current state transition. We estimate this baseline online with a learned critic co-trained alongside the world model; regressing a single scalar, the critic converges well before the world model saturates, redirecting exploration toward learnable transitions without oracle knowledge of the noise floor. The reward is higher for learnable transitions and collapses toward the baseline for stochastic ones, effectively separating epistemic (reducible) from aleatoric (irreducible) prediction error online. Prior prediction-error curiosity formulations, from Schmidhuber (1991) to learned-feature-space variants, emerge as special cases corresponding to specific approximations of this baseline. Experiments on a stochastic grid world show that Curiosity-Critic outperforms prediction-error and visitation-count baselines in convergence speed and final world model accuracy.

Problem

• curiosity
• prediction error
• world model
• intrinsic reward
• epistemic uncertainty

Innovation

• Curiosity-Critic
• intrinsic reward
• world model
• prediction error
• epistemic uncertainty