Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

2026-09-16

AI Summary

研究解决了模型强化学习中动态变化下的重放缓存保留问题，通过衡量变化幅度和数据新鲜度来选择合适的重放策略。

Abstract

Adapting to changes in robot dynamics requires learning from new data without discarding experience that may still be useful. In continual model-based reinforcement learning (RL), replay collected before a dynamics change can slow adaptation, while removing it unnecessarily reduces available training data and can be especially costly if earlier dynamics return. We study when recent transitions are preferable to the full replay history. Two quantities characterize this trade-off: change magnitude and age-staleness area under the curve (AUC), measuring how well transition age separates stale from fresh data. Forgetting stale data helps after large permanent shifts but hurts when dynamics recur and older data becomes useful again. Choosing a replay strategy therefore depends on predicting when older data will help or hurt. We test these effects across two locomotion morphologies, two model-based RL algorithms, and Real-World RL benchmark perturbations. Because ground-truth staleness labels are unavailable on deployed robots, we evaluate whether an estimator built from interaction data can still provide the quantities needed to choose a replay strategy after permanent changes. Our results show that replay retention depends on change magnitude and on how the dynamics evolve.

Problem

• reinforcement learning
• dynamics shift
• replay retention
• adaptation

Innovation

• replay retention
• dynamics shift
• model-based reinforcement learning
• age-staleness AUC