Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehicles

2026-04-09

AI Summary

This work addresses the challenge of reliably and interpretably translating open-ended natural language instructions from passengers into low-level control signals for autonomous vehicles, while ensuring real-time performance and safety. The authors propose a scheduler-centric execution framework that leverages a large language model (LLM) to parse semantic instructions and generate executable scripts, which dynamically orchestrate multiple model predictive control (MPC)-based motion planners. A closed-loop feedback mechanism enables real-time generation of control signals, establishing a transparent decision chain from semantics to actions. By integrating LLMs with a multi-planner scheduling architecture, this approach decouples the timescales of semantic reasoning and vehicle control, yielding a traceable and interpretable system. The study also introduces the first high-fidelity benchmark for evaluating open-ended instruction following. Experiments demonstrate significant improvements in task completion rates, reduced LLM query costs, safety and compliance on par with specialized methods, and strong robustness to LLM inference latency.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Most Human-Machine Interaction (HMI) research overlooks the maneuvering needs of passengers in autonomous driving (AD). Natural language offers an intuitive interface, yet translating passenger open-ended instructions into control signals, without sacrificing interpretability and traceability, remains a challenge. This study proposes an instruction-realization framework that leverages a large language model (LLM) to interpret instructions, generates executable scripts that schedule multiple model predictive control (MPC)-based motion planners based on real-time feedback, and converts planned trajectories into control signals. This scheduling-centric design decouples semantic reasoning from vehicle control at different timescales, establishing a transparent, traceable decision-making chain from high-level instructions to low-level actions. Due to the absence of high-fidelity evaluation tools, this study introduces a benchmark for open-ended instruction realization in a closed-loop setting. Comprehensive experiments reveal that the framework significantly improves task-completion rates over instruction-realization baselines, reduces LLM query costs, achieves safety and compliance on par with specialized AD approaches, and exhibits considerable tolerance to LLM inference latency. For more qualitative illustrations and a clearer understanding.

Problem

• open-ended instruction
• autonomous vehicles
• human-machine interaction
• instruction realization
• natural language interface

Innovation

• LLM-enabled scheduling
• open-ended instruction realization
• multi-planner MPC
• interpretable autonomy
• closed-loop benchmarking