From Pixels to Temporal Correlations: Learning Informative Representations for Reinforcement Learning Pre-training

2026-07-01

AI Summary

Existing unsupervised video pretraining methods overly rely on static pixel information and often neglect subtle yet critical dynamic details, thereby limiting their representational capacity. This work proposes a Temporal Correlation Space with Multi-scale Temporal Contrastive Learning (MTCL) framework that overcomes the limitations of conventional single-step prediction and image reconstruction by explicitly modeling dynamic dependencies among video elements across multiple temporal scales in a balanced manner. Without requiring any labels, the approach effectively captures essential temporal structures, significantly improving both sample efficiency and final performance on downstream reinforcement learning tasks. These results demonstrate the generality and effectiveness of the learned representations.

Abstract

Unsupervised pre-training on large-scale datasets has demonstrated significant potential for improving the sample efficiency and performance of Reinforcement Learning (RL). Given the large-scale action-free internet videos, existing methods utilize single-step transition prediction and image reconstruction to learn representations. However, these methods prefer to preserve large-proportion stationary information in the pixel space, neglecting small but crucial information. To preserve enough information in the representation, it is essential to pay equal attention to each element in videos. Specifically, we propose a temporal correlation space to distinguish each element. For implementation, we introduce the Multi-scale Temporal Contrastive Learning (MTCL) method to model multi-scale temporal correlations separately. This approach can balance the attention of different elements and yield more informative representations, effectively supporting policy learning in various downstream tasks. Experimental results demonstrate that our method improves sample efficiency and asymptotic performance across various downstream tasks.

Problem

• Reinforcement Learning Pre-training
• Unsupervised Representation Learning
• Temporal Correlations
• Sample Efficiency
• Action-free Videos

Innovation

• Temporal Correlation
• Multi-scale Temporal Contrastive Learning
• Unsupervised Pre-training
• Reinforcement Learning
• Informative Representations