When Robots Sleep: Offline Skill Consolidation for Shared-Policy Robot Learning

2026-06-16

AI Summary

This work addresses the degradation of shared policies and declining skill reliability in long-term robot deployment, caused by insufficient historical data and the absence of task-specific modules. To mitigate these issues, the authors propose a wake-sleep continual learning framework: new skills are acquired during an online “wake” phase, while a frozen skill memory is leveraged in an offline “sleep” phase to consolidate the shared policy. The study identifies and tackles the previously unaddressed phenomenon of “skill coupling collapse” through a Nash bargaining-based multi-objective gradient fusion mechanism, augmented with adaptive anchoring and local excitability for stable optimization. Offline policy updates under both reinforcement and imitation learning are enabled via frozen critic/actor snapshots and an unordered replay buffer that constructs differentiable surrogate objectives. Evaluated on Meta-World MT5, the method achieves a 64% average success rate improvement and doubles skill-pair reliability; it also significantly outperforms continual imitation learning baselines on SurgicAI.

Abstract

Robots that learn over long deployments must add new skills without losing the shared policy structure that makes earlier skills reusable. We study sequential robot skill learning, where previous trajectories and task losses may be unavailable, and the deployed policy must remain a single shared controller without task-specific heads, routing, or adapters. We identify skill-coupling collapse, a failure mode in which individual skill success remains non-trivial while reliability among related skills deteriorates. We propose Sleeping Robots, a wake-sleep framework that learns each new skill during wake and consolidates the shared policy offline during sleep using compact frozen skill memories: frozen critics with unordered state buffers for reinforcement learning and frozen actor snapshots with unordered observation buffers for imitation learning. During sleep, these memories define differentiable surrogate objectives whose gradients are combined through Nash bargaining, with adaptive anchoring and local excitability for stable consolidation. On Meta-World MT5, Sleeping Robots improves average success by 64 % and pairwise reliability by x 2.0 over the strongest non-oracle baseline, and on SurgicAI it improves average success and backward transfer relative to continual imitation baselines while remaining competitive on pairwise reliability.

Problem

• sequential robot skill learning
• shared-policy
• skill-coupling collapse
• offline consolidation
• continual learning

Innovation

• offline skill consolidation
• shared-policy learning
• skill-coupling collapse
• wake-sleep framework
• Nash bargaining