Stealthy World Model Manipulation via Data Poisoning

2026-06-17

AI Summary

This work addresses the vulnerability of world model–based reinforcement learning to stealthy data poisoning attacks during fine-tuning and proposes the SWAAP framework. SWAAP employs a two-stage strategy: first constructing a harmful yet imperceptible target world model, then applying constrained gradient matching to perturb only a small subset of transition data, thereby steering the agent toward learning the malicious model while ensuring poisoned samples remain highly consistent with the original data distribution. SWAAP constitutes the first successful stealthy poisoning attack against learned world models, integrating first-order bilevel optimization, the transition gradient theorem, and predictive error regularization. Evaluated across multiple continuous control tasks, it significantly degrades agent performance and effectively evades non-adaptive defenses such as residual-based detection, CUSUM, and TRIM, revealing critical limitations in current detection mechanisms.

Abstract

Model-based learning agents use learned world models to predict future states, plan actions, and adapt to new environments. However, the process of updating world models from collected experience creates a training-time attack surface: adversarially poisoned fine-tuning trajectories can manipulate the learned dynamics and thereby corrupt downstream planning. In this paper, we propose SWAAP, the first two-stage data poisoning framework for learned world models. In the first stage, SWAAP identifies a harmful target world model that induces low-return behavior under planning while remaining close to clean dynamics, using first-order bilevel optimization enabled by a transition-gradient theorem. In the second stage, SWAAP realizes this target through stealth-constrained gradient matching, modifying only a limited fraction of fine-tuning transition targets so that the induced training gradients steer the victim model toward the adversarial target, while a prediction-error regularizer encourages the poisoned targets to remain close to the world model's natural approximation error. To assess attack stealthiness, we evaluate defenses and detectability across three stages of the poisoning pipeline: pre-training detection of poisoned transitions, robust training during fine-tuning, and test-time monitoring of the resulting world model. Across diverse continuous-control tasks, SWAAP causes substantial performance degradation while keeping poisoned transitions close to clean data and evading the evaluated non-adaptive residual/CUSUM/TRIM-style defenses. These results reveal a practical vulnerability in world-model adaptation pipelines and highlight the need for robustness methods that protect both world-model training data and learned dynamics.

Problem

• data poisoning
• world model
• stealthy attack
• model-based reinforcement learning
• adversarial manipulation

Innovation

• data poisoning
• world models
• stealthy attack
• bilevel optimization
• gradient matching