Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLA

2026-04-30

AI Summary

This work addresses the degraded zero-shot transfer performance of autonomous driving agents in unseen towns by proposing an enhanced approach based on the Dreamer latent world model. The method introduces two training-phase-only auxiliary mechanisms—multi-step visual-semantic embedding prediction and town-adversarial regularization—to improve generalization to novel town layouts without relying on navigation instructions or maps. It explicitly disentangles causal contextual features from standard control features and employs semantic rollout supervision to enhance cross-scenario consistency. Evaluated on the CARLA simulator using Town03 and Town04 under fixed routes, no traffic interference, and constant weather conditions, the proposed method achieves the highest average task success rate among Dreamer-based approaches, demonstrating its effectiveness.

Abstract

Learned driving agents often degrade when deployed in unseen environments. This paper studies a deliberately bounded instance of that problem in the CARLA simulator: zero-shot transfer of a closed-loop fixed-route driving agent from Town05 and Town06 to unseen Town03 and Town04. The study isolates structural town shift by keeping weather fixed to ClearNoon and removing traffic and pedestrians. We build on a Dreamer-style latent world-model agent and add two training-only auxiliary losses: multi-horizon prediction of future visual-semantic embeddings along imagined rollouts and town-adversarial supervision on a semantic projection of the recurrent latent state. A causal context feature conditions the semantic rollout predictor, while the actor and critic retain the standard control feature. The policy receives no navigation command, route polyline, goal pose, or map input; the reference route is used only by the environment for reward, progress, success, and termination. Across the evaluated held-out towns, the proposed model achieves the highest mean success rate among the included Dreamer-family methods. Secondary safety and lane-keeping metrics are mixed across towns. These results support a bounded conclusion: in this controlled fixed-weather CARLA setting, semantic rollout supervision combined with town-adversarial regularization improves mean held-out-town route completion.

Problem

• zero-shot transfer
• domain shift
• autonomous driving
• CARLA simulator
• fixed-route driving

Innovation

• semantic rollout
• town-adversarial regularization
• zero-shot transfer
• latent world model
• fixed-route driving