KnowDiffuser: A Knowledge-Guided Diffusion Planner with LM Reasoning and Prior-Informed Trajectory Initialization

2026-03-11

AI Summary

This work addresses the semantic-physical disconnect in existing autonomous driving planners, where language models struggle to generate physically feasible trajectories and diffusion models lack semantic reasoning capabilities. To bridge this gap, the authors propose a novel framework that tightly couples high-level semantic reasoning with low-level trajectory generation. Specifically, a language model performs context-aware meta-action reasoning and maps its output to a prior trajectory that guides the diffusion process. A two-stage truncated denoising mechanism is introduced to efficiently produce trajectories that are both semantically consistent and dynamically feasible. Evaluated on the nuPlan benchmark, the method significantly outperforms existing planners, achieving state-of-the-art performance in both open-loop and closed-loop settings, thereby effectively reconciling semantic intent with physical realizability.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Multiagent Planning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advancements in Language Models (LMs) have demonstrated strong semantic reasoning capabilities, enabling their application in high-level decision-making for autonomous driving (AD). However, LMs operate over discrete token spaces and lack the ability to generate continuous, physically feasible trajectories required for motion planning. Meanwhile, diffusion models have proven effective at generating reliable and dynamically consistent trajectories, but often lack semantic interpretability and alignment with scene-level understanding. To address these limitations, we propose \textbf{KnowDiffuser}, a knowledge-guided motion planning framework that tightly integrates the semantic understanding of language models with the generative power of diffusion models. The framework employs a language model to infer context-aware meta-actions from structured scene representations, which are then mapped to prior trajectories that anchor the subsequent denoising process. A two-stage truncated denoising mechanism refines these trajectories efficiently, preserving both semantic alignment and physical feasibility. Experiments on the nuPlan benchmark demonstrate that KnowDiffuser significantly outperforms existing planners in both open-loop and closed-loop evaluations, establishing a robust and interpretable framework that effectively bridges the semantic-to-physical gap in AD systems.

Problem

• autonomous driving
• motion planning
• semantic reasoning
• trajectory generation
• diffusion models

Innovation

• knowledge-guided diffusion
• language model reasoning
• trajectory initialization
• semantic-to-physical planning
• two-stage denoising