OVITA: Open-Vocabulary Interpretable Trajectory Adaptations

2025-08-24

AI Summary

Non-expert users struggle to naturally modulate robot trajectories in dynamic, unstructured environments. Method: This paper proposes a natural-language-instruction-based trajectory adaptation framework. It leverages large language models (LLMs) to compile open-vocabulary instructions into executable and interpretable Python code that directly modifies waypoints generated by motion planners; a second LLM automatically parses the generated code to enhance decision transparency. The framework unifies demonstration learning and natural language understanding through code as a unified policy interface, decoupling semantic interpretation from low-level motion control. Contribution/Results: Evaluated on KUKA iiwa, Jackal, and CrazyFlie platforms in both simulation and real-world experiments, the framework enables real-time trajectory re-planning, demonstrates cross-task and cross-platform generalization, and significantly improves non-expert users’ operational efficiency and trust.

Technology Category

• Intelligent Robots: Motion and Path Planning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Adapting trajectories to dynamic situations and user preferences is crucial for robot operation in unstructured environments with non-expert users. Natural language enables users to express these adjustments in an interactive manner. We introduce OVITA, an interpretable, open-vocabulary, language-driven framework designed for adapting robot trajectories in dynamic and novel situations based on human instructions. OVITA leverages multiple pre-trained Large Language Models (LLMs) to integrate user commands into trajectories generated by motion planners or those learned through demonstrations. OVITA employs code as an adaptation policy generated by an LLM, enabling users to adjust individual waypoints, thus providing flexible control. Another LLM, which acts as a code explainer, removes the need for expert users, enabling intuitive interactions. The efficacy and significance of the proposed OVITA framework is demonstrated through extensive simulations and real-world environments with diverse tasks involving spatiotemporal variations on heterogeneous robotic platforms such as a KUKA IIWA robot manipulator, Clearpath Jackal ground robot, and CrazyFlie drone.

Problem

• Adapting robot trajectories to dynamic situations and user preferences
• Enabling non-expert users to adjust trajectories through natural language
• Providing interpretable trajectory adaptations across heterogeneous robotic platforms

Innovation

• Open-vocabulary language-driven framework for trajectory adaptation
• Leverages multiple pre-trained LLMs to integrate user commands
• Employs code as adaptation policy for waypoint adjustments