Augmentations for Robust and Efficient Imitation Learning in Streamed Video Games

2026-07-15

AI Summary

This work addresses covariate shift in streaming video games caused by visual artifacts introduced by network latency and compression. To mitigate this issue, the authors propose a spatiotemporal data augmentation method tailored for streaming environments, which simulates common artifacts such as blocky pixels, global blur, and ghosting effects. The augmentation is integrated into a Predictive Inverse Dynamics Model (PIDM) framework, where training occurs in latent space by jointly conditioning on future states and inverse dynamics policies. As the first augmentation strategy specifically designed for streaming gameplay, the approach substantially enhances agent robustness under low-bandwidth and high-latency conditions: it achieves a 41% performance gain in stable environments under identical data budgets, and when subjected to network latency, exhibits only a 7.45% performance drop—significantly outperforming baseline models, which suffer a 49.82% decline.

Abstract

Imitation learning is an appealing way to scale game-playing agents to complex 3D environments by training policies to map visual observations to actions from human demonstrations. However, these demonstrations are expensive to collect and modern game-playing is often done through streaming in which network delay and compression introduce spatiotemporally correlated visual artifacts that can cause a covariance shift at test time. To address these challenges, we propose streaming augmentations that mimic four types of artifacts commonly encountered during streaming with low-bandwidth network connection: pixelated blocks and scrubs, global blur, and ghosting. We instantiate our approach on top of predictive inverse dynamics models (PIDM), which combine future-state conditioning with an inverse dynamics policy in a learned latent space, and evaluate the impact of our augmentations across three tasks in modern 3D video games. Under stable streaming conditions, agents trained with spatiotemporal augmentations achieve up to 41% higher evaluation performance compared to agents trained without augmentations under an identical data budget. When network lag is introduced, agents trained with augmentations degrade by only 7.45% vs 49.82% of the original performance for agents trained only with the original data. These results clearly indicate that spatiotemporal augmentations tailored for the streaming setting are a simple yet powerful tool to train robust and efficient game-playing agents.

Problem

• imitation learning
• streaming video games
• visual artifacts
• covariance shift
• spatiotemporal augmentations

Innovation

• streaming augmentations
• spatiotemporal artifacts
• imitation learning
• predictive inverse dynamics models
• robustness