SHRIMP: Iterative Refinement of Robot Task Plans

2026-08-09

AI Summary

Natural language instructions often lead to unreliable robot task planning due to semantic ambiguity, and plans generated by large language models (LLMs) typically lack transparency, undermining user trust. This work proposes the first system that enables users to interactively refine robot task plans through natural language: it parses instructions into hierarchical robot primitive plans, supports iterative correction via re-prompting or explicit editing, and validates each revised plan in simulation before deployment on a physical robot. By integrating LLMs, hierarchical planning, simulation-based verification, and interactive refinement, the system establishes an interpretable and editable mapping from natural language to executable actions. A user study with 35 participants performing tabletop kitchen tasks demonstrates that this approach significantly enhances users’ sense of control and perceived system transparency.

Abstract

As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification suffers from semantic ambiguity, and generative models lack transparency for how language instructions become robot actions, making it difficult for users to validate the plan before execution. To address these issues, we introduce SHRIMP, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (n=35), we validate that SHRIMP improves perceived control and enhances robot transparency. System videos and source code are available at https://wisc-hci.github.io/SHRIMP.

Problem

• natural language
• semantic ambiguity
• robot task planning
• transparency
• human-robot interaction

Innovation

• iterative refinement
• natural language programming
• robot task planning
• human-robot interaction
• large language models