VLM-MPPI: Grounding Natural Language in Behaviorally Diverse Trajectories for Aerial Navigation

2026-09-16

AI Summary

研究提出VLM-MPPI框架，通过结合视觉-语言模型和行为条件的MPPI规划器，解决在复杂室内环境中无人机根据自然语言指令进行导航的问题。

Abstract

We present a hierarchical UAV navigation framework that aligns natural-language intent with dynamically feasible flight behaviors in cluttered indoor environments. To bridge the gap between abstract semantics and low-level control, we employ a parallelized ensemble of six behavior-conditioned Model Predictive Path Integral (MPPI) planners. Crucially, by designing mode-specific guiding costs and sampling biases, we induce distinct trajectory modes that converge to unique behavioral means, yielding a compact set of intentionally diverse candidates rather than mere stochastic variations. We project these 3D candidates onto the onboard first-person-view RGB stream, turning language grounding into a visual action selection problem. A pretrained vision--language model (VLM) asynchronously selects the candidate index given the overlaid FPV image and a natural-language prompt, while MPPI replans at 20Hz and a PID-based low-level controller tracks the selected trajectory. We implement the full pipeline in NVIDIA Isaac Sim and on a real-world quadrotor platform equipped with LiDAR and RGB sensing. Experiments in both simulation and real-world flights show semantically meaningful behavior diversity, robust language alignment despite VLM latency, and safe, repeatable flight across all modes, achieving 100% task success in our evaluated scenarios.

Problem

• natural-language intent
• dynamically feasible flight behaviors
• cluttered indoor environments
• UAV navigation

Innovation

• behavior-conditioned MPPI planners
• mode-specific guiding costs
• visual action selection problem
• vision--language model (VLM)