IROSA: Interactive Robot Skill Adaptation using Natural Language

2026-03-04

AI Summary

This work proposes a tool-oriented framework to address the challenge of enabling robotic systems in industrial settings to adapt flexibly and safely to new tasks without fine-tuning. By leveraging a pre-trained large language model (LLM), the approach selects and parameterizes specialized skill tools through natural language instructions, achieving open-vocabulary skill adaptation. A protective abstraction layer decouples the LLM from low-level hardware, ensuring safety, interpretability, and transparency. Evaluated on a 7-degree-of-freedom force-controlled robot platform in conjunction with imitation learning, the system successfully executes natural language–guided skill adjustments—including speed modulation, trajectory correction, and obstacle avoidance—in a bearing ring insertion task, demonstrating the method’s effectiveness and practicality.

Technology Category

• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Manipulation
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.

Problem

• robot skill adaptation
• natural language
• foundation models
• industrial robotics
• open-vocabulary

Innovation

• Interactive Robot Skill Adaptation
• Natural Language
• Foundation Models
• Tool-based Architecture
• Imitation Learning