Lifelong Language-Conditioned Robotic Manipulation Learning

2026-03-05

AI Summary

This work addresses the challenge of catastrophic forgetting in language-guided robotic agents when continuously learning new manipulation skills, which hinders their deployment in dynamic environments. To mitigate this issue, the authors propose SkillsCrafter, a novel framework that constructs a skill semantic subspace via singular value decomposition and integrates cross-skill similarity aggregation with skill specialization mechanisms. This approach effectively preserves knowledge of previously acquired skills while enabling the sharing of commonalities across skills, thereby supporting both continual learning and generalization. Experimental results demonstrate that SkillsCrafter significantly outperforms existing baselines in multi-skill continual learning tasks, achieving superior performance in both retention of old skills and generalization to new ones.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Life-Long and Continual Learning
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Traditional language-conditioned manipulation agent sequential adaptation to new manipulation skills leads to catastrophic forgetting of old skills, limiting dynamic scene practical deployment. In this paper, we propose SkillsCrafter, a novel robotic manipulation framework designed to continually learn multiple skills while reducing catastrophic forgetting of old skills. Specifically, we propose a Manipulation Skills Adaptation to retain the old skills knowledge while inheriting the shared knowledge between new and old skills to facilitate learning of new skills. Meanwhile, we perform the singular value decomposition on the diverse skill instructions to obtain common skill semantic subspace projection matrices, thereby recording the essential semantic space of skills. To achieve forget-less and generalization manipulation, we propose a Skills Specialization Aggregation to compute inter-skills similarity in skill semantic subspaces, achieving aggregation of the previously learned skill knowledge for any new or unknown skill. Extensive experiments demonstrate the effectiveness and superiority of our proposed SkillsCrafter.

Problem

• catastrophic forgetting
• lifelong learning
• language-conditioned manipulation
• robotic manipulation
• skill adaptation

Innovation

• lifelong learning
• catastrophic forgetting
• language-conditioned manipulation
• semantic subspace
• skill aggregation