Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Driving

2026-07-11

AI Summary

This work addresses the challenges of trajectory planning in autonomous driving within complex urban environments, where limited coverage of long-tail interactive scenarios, weak consistency with downstream constraints, and insufficient exploitation of high-level semantic information hinder performance. To tackle these issues, the authors propose a differentiable trajectory planning framework integrated with a large language model (LLM). The approach enhances the training distribution through agent-centric data augmentation, incorporates a complexity-aware asynchronous LLM module for low-overhead extraction of high-level semantics, and enables end-to-end gradient backpropagation via differentiable optimization. Evaluated on the nuPlan Hard20 benchmark, the method achieves state-of-the-art composite scores of 83.63 and 78.29 in non-reactive and reactive settings, respectively, and demonstrates real-time closed-loop deployment capability validated on CARLA-ROS.

Abstract

Autonomous driving planning is a key component of IoT-enabled intelligent transportation systems, requiring vehicles to generate safe, efficient, and executable trajectories in complex urban environments from multi-source contextual information. While imitation learning (IL) has shown promise on large-scale datasets, IL-based planners still suffer from limited coverage of complex long-tail interactions, weak consistency with downstream constrained refinement, and insufficient use of high level scene semantics under real time constraints. To address these issues, this paper proposes a large language model (LLM) enhanced differentiable trajectory planning framework for IoT-enabled autonomous driving. Specifically, we introduce a surrounding agent centric data augmentation strategy to reorganize sur rounding agent trajectories as additional planning supervision, thereby improving the training distribution without collecting additional raw data. We further design a complexity-aware asyn chronous LLM-based semantic enhancement module to extract scene-related high-level semantic features with controlled online overhead. In addition, a differentiable optimization module is incorporated to refine generated trajectories with explicit residual penalties while backpropagating optimization gradients to the upstream planner. Experiments show that the proposed method achieves the best overall scores of 83.63 and 78.29 on the nuPlan closed-loop nonreactive and reactive Hard20 benchmarks, respectively, and CARLA-ROS tests further verify its online deployment and real time closed-loop execution capability.

Problem

• autonomous driving
• trajectory planning
• long-tail interactions
• semantic understanding
• real-time constraints

Innovation

• Large Language Model
• Differentiable Trajectory Planning
• Data Augmentation
• Semantic Enhancement
• Autonomous Driving