🤖 AI Summary
To address the low sample efficiency of reinforcement learning (RL), this paper proposes a self-supervised disentanglement method grounded in egomotion-aware visual prediction. By modeling the agent’s own motion as a proxy signal, the approach enables unsupervised disentanglement of agent-centric dynamics from environmental dynamics—without requiring supervision masks or explicit segmentation. Visual motion prediction is incorporated as an implicit regularizer to constrain policy learning. This work is the first to integrate human-inspired motion prediction mechanisms into RL frameworks, synergistically combining egomotion modeling, cross-modal temporal consistency learning, and model-agnostic RL algorithms (PPO and SAC). Empirical validation on real robotic platforms demonstrates that motion-invariant visual prediction significantly improves training efficiency by 23% on average and enhances task performance by 8%.
📝 Abstract
Despite the significant advancements in Deep Reinforcement Learning (RL) observed in the last decade, the amount of training experience necessary to learn effective policies remains one of the primary concerns both in simulated and real environments. Looking to solve this issue, previous work has shown that improved training efficiency can be achieved by separately modeling agent and environment, but usually requiring a supervisory agent mask. In contrast to RL, humans can perfect a new skill from a very small number of trials and in most cases do so without a supervisory signal, making neuroscientific studies of human development a valuable source of inspiration for RL. In particular, we explore the idea of motor prediction, which states that humans develop an internal model of themselves and of the consequences that their motor commands have on the immediate sensory inputs. Our insight is that the movement of the agent provides a cue that allows the duality between agent and environment to be learned. To instantiate this idea, we present Ego-Foresight, a self-supervised method for disentangling agent and environment based on motion and prediction. Our main finding is that visuomotor prediction of the agent provides regularization to the RL algorithm, by encouraging the actions to stay within predictable bounds. To test our approach, we first study the ability of our model to visually predict agent movement irrespective of the environment, in real-world robotic interactions. Then, we integrate Ego-Foresight with a model-free RL algorithm to solve simulated robotic manipulation tasks, showing an average improvement of 23% in efficiency and 8% in performance.