Bootstrapping Human-Like Planning via LLMs

2025-06-27

AI Summary

Task programming for robotic end-users faces a trade-off between natural language (intuitive yet ambiguous) and drag-and-drop interfaces (precise yet inefficient). This paper proposes an LLM-driven approach that synergistically integrates both paradigms, establishing an end-to-end generation pipeline that maps natural language instructions to structured, fine-grained, human-like action sequences. Our key contributions are a semantic alignment mechanism and an action granularity control strategy—ensuring outputs are both interpretable and execution-accurate. Evaluated on a manually annotated robotic task dataset, our method significantly outperforms baseline approaches. Large language models yield superior generation quality, while even smaller models achieve practical utility, demonstrating the framework’s effectiveness, generalizability, and deployment feasibility.

Technology Category

• Intelligent Robots: Manipulation
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Robot end users increasingly require accessible means of specifying tasks for robots to perform. Two common end-user programming paradigms include drag-and-drop interfaces and natural language programming. Although natural language interfaces harness an intuitive form of human communication, drag-and-drop interfaces enable users to meticulously and precisely dictate the key actions of the robot's task. In this paper, we investigate the degree to which both approaches can be combined. Specifically, we construct a large language model (LLM)-based pipeline that accepts natural language as input and produces human-like action sequences as output, specified at a level of granularity that a human would produce. We then compare these generated action sequences to another dataset of hand-specified action sequences. Although our results reveal that larger models tend to outperform smaller ones in the production of human-like action sequences, smaller models nonetheless achieve satisfactory performance.

Problem

• Combining natural language and drag-and-drop interfaces for robot task specification
• Generating human-like action sequences from natural language using LLMs
• Comparing LLM-generated action sequences with hand-specified sequences

Innovation

• Combines natural language and drag-and-drop interfaces
• Uses LLM pipeline for human-like action sequences
• Compares generated and hand-specified sequences