Large Language Model-Driven Closed-Loop UAV Operation with Semantic Observations

2025-07-02

AI Summary

Large language models (LLMs) exhibit weak logical reasoning, poor fault tolerance, and a disconnect between state feedback and decision-making when applied to autonomous unmanned aerial vehicle (UAV) control. Method: This paper proposes a semantic dual-module closed-loop framework: it maps real-time numerical UAV states to natural-language trajectory descriptions and leverages the LLM as both a code generator and an evaluator, enabling an internal “generate–execute–evaluate–refine” loop within simulation. The framework integrates semantic observation transformation and text-based state feedback mechanisms. Contribution/Results: This approach significantly improves reasoning consistency and operational robustness under complex high-level instructions. Experiments across multiple high-complexity flight tasks demonstrate substantial gains in task success rate and completion rate over baseline methods; notably, performance remains consistently superior as task complexity increases.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models (LLMs) have revolutionized robotic autonomy, including Unmanned Aerial Vehicles (UAVs). Recent studies have demonstrated the potential of LLMs for translating human instructions into executable control code for UAV operations. However, LLMs still face challenges from logical reasoning and complex decision-making, leading to concerns about the reliability of LLM-driven UAV operations. In this paper, we propose a LLM-driven closed-loop control framework that enables reliable UAV operations powered by effective feedback and refinement using two LLM modules, i.e., a Code Generator and an Evaluator. Our framework transforms numerical state observations from UAV operations into natural language trajectory descriptions to enhance the evaluator LLM's understanding of UAV dynamics for precise feedback generation. Our framework also enables a simulation-based refinement process, and hence eliminates the risks to physical UAVs caused by incorrect code execution during the refinement. Extensive experiments on UAV control tasks with different complexities are conducted. The experimental results show that our framework can achieve reliable UAV operations using LLMs, which significantly outperforms baseline approaches in terms of success rate and completeness with the increase of task complexity.

Problem

• Enhancing UAV operation reliability using LLM-driven closed-loop control
• Improving LLM logical reasoning for complex UAV decision-making tasks
• Reducing physical UAV risks via simulation-based code refinement

Innovation

• LLM-driven closed-loop UAV control framework
• Natural language trajectory descriptions for feedback
• Simulation-based refinement to avoid physical risks