Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Models

2025-04-06

AI Summary

Existing foundation models for autonomous vehicle trajectory planning suffer from high hallucination rates, poor uncertainty quantification, excessive inference latency, and insufficient integration of traffic safety principles. This paper proposes LetsPi, the first framework adopting a two-stage physics–knowledge co-design paradigm: “memory construction + rapid inference.” In Stage I, a retrievable and evolvable memory bank is constructed from real-world driving experience. In Stage II, a large language model (LLM) is tightly integrated with a physics-driven social force model (SFM), augmented by physics-aware prompting and traffic safety metrics—including time-to-collision (TTC) and deceleration required to avoid collision (DRAC). Evaluated on the HighD dataset, LetsPi achieves superior performance across five safety metrics (e.g., collision risk, emergency braking frequency), reduces inference latency by 47%, and lowers hallucination rate by over 60%, significantly enhancing both safety compliance and human-like driving behavior.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Foundation models have demonstrated strong reasoning and generalization capabilities in driving-related tasks, including scene understanding, planning, and control. However, they still face challenges in hallucinations, uncertainty, and long inference latency. While existing foundation models have general knowledge of avoiding collisions, they often lack transportation-specific safety knowledge. To overcome these limitations, we introduce LetsPi, a physics-informed, dual-phase, knowledge-driven framework for safe, human-like trajectory planning. To prevent hallucinations and minimize uncertainty, this hybrid framework integrates Large Language Model (LLM) reasoning with physics-informed social force dynamics. LetsPi leverages the LLM to analyze driving scenes and historical information, providing appropriate parameters and target destinations (goals) for the social force model, which then generates the future trajectory. Moreover, the dual-phase architecture balances reasoning and computational efficiency through its Memory Collection phase and Fast Inference phase. The Memory Collection phase leverages the physics-informed LLM to process and refine planning results through reasoning, reflection, and memory modules, storing safe, high-quality driving experiences in a memory bank. Surrogate safety measures and physics-informed prompt techniques are introduced to enhance the LLM's knowledge of transportation safety and physical force, respectively. The Fast Inference phase extracts similar driving experiences as few-shot examples for new scenarios, while simplifying input-output requirements to enable rapid trajectory planning without compromising safety. Extensive experiments using the HighD dataset demonstrate that LetsPi outperforms baseline models across five safety metrics.See PDF for project Github link.

Problem

• Overcoming hallucinations and uncertainty in trajectory planning
• Integrating transportation-specific safety knowledge into foundation models
• Balancing reasoning and computational efficiency in trajectory generation

Innovation

• Physics-informed dual-phase framework for safety
• LLM reasoning with social force dynamics
• Memory bank stores high-quality driving experiences