Temporal Self-Imitation Learning

2026-06-17

AI Summary

This work addresses the inefficiency and behavioral forgetting commonly observed in long-horizon robotic manipulation policies that rely on dense reward signals. To overcome these limitations, the authors propose a time-efficiency-based self-supervised signal that identifies high-performing successful trajectories generated during training. This signal enables an efficiency-weighted self-imitation learning mechanism, augmented with a conditionally adaptive temporal goal. Evaluated across 15 long-horizon manipulation tasks, the proposed approach substantially improves policy sample efficiency, task completion speed, reproducibility of high-efficiency behaviors, and overall training robustness.

Abstract

Long-horizon robot manipulation policies trained with reward shaping can still exploit dense rewards through inefficient interaction, while rare efficient behaviors may be forgotten during training. We argue that temporal efficiency itself provides a powerful and underutilized source of self-supervision for reinforcement learning. We introduce Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework that mines temporally efficient successful trajectories generated during learning and converts them into reusable supervision for future policy improvement. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviors through efficiency-weighted self-imitation learning. Across 15 distinct long-horizon manipulation tasks, TSIL consistently improves learning efficiency, task-completion efficiency, revisitation of fast successful behaviors, and robustness to unstable training conditions. More broadly, our results suggest that the temporal structure of successful behavior itself provides a scalable self-supervisory signal for reinforcement learning beyond manually engineered reward shaping alone.

Problem

• long-horizon manipulation
• reward shaping
• temporal efficiency
• reinforcement learning
• efficient behavior

Innovation

• Temporal Self-Imitation Learning
• reinforcement learning
• long-horizon manipulation
• self-supervision
• efficiency-weighted imitation