When Pre-trained Visual Representations Fall Short: Limitations in Visuo-Motor Robot Learning

2025-02-05

AI Summary

Pre-trained visual representations (PVRs) suffer from poor generalization in embodied robot visuomotor policy learning due to temporal entanglement and sensitivity to scene perturbations. This work presents the first systematic diagnosis of their structural deficiencies in closed-loop control. To address these, we propose a temporal decoupling enhancement mechanism—integrating time-aware modeling with task-completion signals—and a local task-relevant feature selection strategy. Our approach embeds temporal modeling modules and learnable attention gating into masked self-supervised vision encoders (e.g., MAE, BEiT), enabling feature-level spatiotemporal decoupling and out-of-distribution generalization. Evaluated on multiple real-robot manipulation benchmarks, our method achieves an average policy performance improvement exceeding 35% and establishes new state-of-the-art robustness under challenging cross-illumination and occlusion conditions.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Representation Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The integration of pre-trained visual representations (PVRs) into visuo-motor robot learning has emerged as a promising alternative to training visual encoders from scratch. However, PVRs face critical challenges in the context of policy learning, including temporal entanglement and an inability to generalise even in the presence of minor scene perturbations. These limitations hinder performance in tasks requiring temporal awareness and robustness to scene changes. This work identifies these shortcomings and proposes solutions to address them. First, we augment PVR features with temporal perception and a sense of task completion, effectively disentangling them in time. Second, we introduce a module that learns to selectively attend to task-relevant local features, enhancing robustness when evaluated on out-of-distribution scenes. Our experiments demonstrate significant performance improvements, particularly in PVRs trained with masking objectives, and validate the effectiveness of our enhancements in addressing PVR-specific limitations.

Problem

• Limitations of pre-trained visual representations
• Temporal entanglement in robot learning
• Robustness to scene perturbations

Innovation

• Augments PVR with temporal perception
• Introduces selective attention module
• Enhances robustness in out-of-distribution scenes