SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming

2026-08-14

AI Summary

This study addresses the challenges of generating code for complex tasks and poor skill reusability in natural language robot programming by proposing an interactive programming system. The approach employs a generate-and-test architecture to iteratively refine code, integrated with an online library learning algorithm that compresses successful sequences into reusable macro skills, thereby enabling continuous program abstraction. Experimental evaluations in manipulation and caregiving domains demonstrate that the system significantly improves code generation success rates and usability while effectively reducing user programming burden. These results validate the efficacy of evaluation-guided generation and dynamic skill library construction in enhancing programming efficiency for complex robotic tasks.

Abstract

Natural-language interfaces can lower the barrier to programming robots, but existing systems struggle when users request complex tasks. While large language models (LLMs) perform well with simple commands, they often struggle to generate code for multi-step tasks, decompose high-level instructions, or reuse prior solutions. We present SkillComposer, an interactive natural-language robot programming system for simulation environments that continually learns reusable program abstractions. SkillComposer uses a generate-test architecture in which an LLM iteratively generates and revises robot programs before execution. Successful programs are stored and processed by an online library-learning algorithm that compresses recurring function sequences into reusable macro skills for future tasks. We evaluate SkillComposer through ablation experiments and a user study with 12 participants to determine its effectiveness on manipulation and robot caregiving tasks. The results show that evaluator-guided generation and learned abstractions improve success rates and usability while reducing user effort in natural-language robot programming.

Problem

• Natural-Language Robot Programming
• Complex Tasks
• Large Language Models
• Skill Reuse
• Task Decomposition

Innovation

• Reusable Skills
• Library Learning
• Generate-Test Architecture
• Natural-Language Robot Programming
• Program Abstractions