Concept-Guided Spatial Regularization for World Models in Atari Pong

2026-07-16

AI Summary

Existing visual world models inadequately capture task-critical concepts such as the ball in Atari Pong, leading to visual and dynamical distortions during closed-loop rollouts and substantially degrading zero-shot model-based reinforcement learning (MBRL) performance in pixel space. To address this limitation, this work proposes Concept-Guided Spatial Regularization (CGSReg), which enforces auxiliary pixel-wise reconstruction constraints on key regions identified via semantic segmentation to enhance the model’s fidelity to core objects. Experimental results demonstrate that CGSReg significantly improves both rollout quality and zero-shot policy performance across representative architectures—including DreamerV3, DIAMOND, and TWISTER—effectively narrowing the performance gap between simulated and real-world environments.

Abstract

World models are usually evaluated as components of model-based reinforcement learning (MBRL) systems, while the world models themselves are rarely studied in isolation. We examine five representative visual world-model agents in Atari Pong: DreamerV3, DIAMOND, TWISTER, Simulus, and STORM. After reproducing their training pipelines and matching the reported agent performance, we freeze the learned world models and evaluate them with a closed-loop rollout diagnostic: a policy trained separately from the corresponding MBRL agent interacts with each frozen model, and the generated video trajectories are inspected for visual and dynamical errors. Across all five models, the rollouts contain clear failures, including ball disappearance, incorrect ball motion, and invalid ball-paddle interactions. Beyond visual trajectories, we further evaluate them with pixel-space zero-shot MBRL, where a new policy is trained entirely inside a frozen world model and then evaluated in the real environment. Across all five models, the resulting policies substantially underperform those produced by the corresponding original MBRL training pipelines. The gap is particularly large for DreamerV3, whose mean return drops from -5.5 to -20.9, near the minimum Pong return of -21. We hypothesize that insufficient modeling of task-critical concepts, such as the ball in Pong, may contribute to these failures. We therefore propose Concept-Guided Spatial Regularization (CGSReg), an auxiliary pixel reconstruction loss applied to segmented concept regions. Experiments show that CGSReg improves both closed-loop rollouts and pixel-space zero-shot MBRL in DreamerV3, DIAMOND, and TWISTER. Its effects vary across the remaining models and evaluation metrics, indicating that CGSReg alone does not address all world-model bottlenecks.

Problem

• world models
• model-based reinforcement learning
• concept representation
• spatial regularization
• Atari Pong

Innovation

• Concept-Guided Spatial Regularization
• World Models
• Model-Based Reinforcement Learning
• Closed-Loop Rollout
• Pixel-Space Zero-Shot MBRL